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I. INTRODUCTION 

The American Recovery and Reinvestment Act of 2009 (ARRA) provided an unprecedented 
amount of federal funds for education in an effort to lessen the effects of the nation’s economic 
downturn and to make a lasting investment in education. Through $97.4 billion in ARRA funds, 
the federal government sought to save education jobs, fund a new wave of innovation in 
education, and support comprehensive efforts to turn around low-performing schools. Race to the 
Top (RTT), a new competitive state grant program authorized under ARRA, received $4.35 
billion to spur states to fund a broad array of policies and practices designed to affect all levels of 
public school education with the ultimate goal of improving student outcomes. The six primary 
topic areas described in the RTT application criteria were (1) improving state capacity to support 
school improvement efforts; (2) adopting standards and assessments that prepare students to 
succeed in college and the workplace; (3) building state data systems that measure student 
growth and success and infonn school staff about how they can improve instruction; 
(4) recruiting, developing, rewarding, and retaining effective teachers and principals; (5) turning 
around low-performing schools; and (6) encouraging conditions in which charter schools can 
succeed. 

To leam about the RTT program, the U.S. Department of Education’s (ED’s) Institute of 
Education Sciences commissioned an evaluation of the program. The RTT evaluation is based on 
a descriptive analysis of state-level education policies and practices, and a descriptive analysis of 
student outcomes at the state level (the latter of which will be presented in a future report), in 
states that received RTT grants compared to states that did not. 

At the request of ED’s Office of English Language Acquisition (OELA), part of the 
evaluation also focuses on how states have addressed the needs of English language learners 
(ELLs) as they used the policies and practices promoted by RTT. ELLs are of particular interest 
to this evaluation because: (1) they are historically lower- achieving than non-ELLs 1 , and (2) the 
RTT program placed particular emphasis on prioritizing the academic achievement of high-needs 
students, including ELLs (U.S. Department of Education 2010). 

The study is being conducted by Mathematica Policy Research, the American Institutes for 
Research, and Social Policy Research Associates. In this chapter we provide background 
infonnation about the RTT program, present prior research on RTT, and provide an overview of 
our evaluation and the contents of this report. 

A. Scope, Purpose, Timing, and Size of RTT Grants 

RTT was designed to promote systemic changes to the U.S. educational system, with the 
ultimate goal of improving student achievement. ED awarded three types of RTT grants between 
2010 and 2013: Round 1, 2, and 3 grants funded by ARRA; RTT-Early Learning Challenge 
grants; and RTT-District grants. 2 This evaluation focuses on the Round 1, 2, and 3 grants to 


1 Since 2002, ELLs’ reading test scores have been below those of non-ELLs on the National Assessment of 
Educational Progress test (National Center for Education Statistics. The Condition of Education. Accessed February 
17, 2014 at https://nces.ed.gov/programs/coe/indicator_cgf.asp). 

2 RTT-Early Learning Challenge grants totaling $913 million were made to 20 states in 2011, 2012, and 2013, 
with the goal of improving early learning and development programs so that children entering kindergarten will be 
ready to succeed. The program focuses on improving early learning and development for young children in two 
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states that were funded by ARRA. In the remainder of this report, when we refer to RTT, we are 
referencing the Round 1 to 3 RTT grants and not the other two types. 

RTT aims to facilitate policies and practices in six areas (listed above). RTT grants were 
designed to encourage and reward states that had strong records of using these policies and 
practices and that presented plans in their RTT applications for furthering policies and practices 
in these areas (U.S. Department of Education 2010). 

The Race to the Top Executive Summary 3 summarizes the program’s priorities and theory 
of action. The general theory of action underlying RTT is that policy changes at the state level 
represent the first step in the process of changing the education system and that the ultimate goal 
is improving student outcomes. Changes would be expected to occur at the state level before 
occurring at lower levels, such as districts, schools, and classrooms. 

RTT grants were awarded through a competition open to all states, the District of Columbia, 
and Puerto Rico. To maximize the program’s likelihood of fostering systemic educational 
policies and practices, ED required states applying for RTT grants to demonstrate substantial 
commitment to implement their proposed plans at the local level. In their RTT applications, 
states had to identify the local education agencies (LEAs) committing to implement all or the 
vast majority of the policies and practices proposed by the state — that is, participating LEAs. In 
contrast, involved LEAs were those committing to participate in specific portions of RTT 
policies and practices that would necessitate statewide or nearly statewide implementation (for 
example, adopting common standards). ED required RTT states to give no less than half of their 
grant funds to participating LEAs, apportioned to them according to the LEAs’ share of Title I 
funds. 4 RTT states were allowed substantial flexibility regarding the use of the balance of their 
grants. They could use the remaining funds to support statewide activities (such as enhancements 
to data, evaluation, or support systems), as supplementary awards to participating LEAs, as sub- 
grants to involved LEAs, or for other activities. 

Across the three rounds of competition, 46 states and the District of Columbia applied for 
RTT grants. (The exceptions were Alaska, North Dakota, Texas, and Vermont.) The timing of 
awards, number of states selected, range of award amounts, and other descriptive information 
about each round are shown in Table 1.1. States’ award amounts varied based on their share of 
the nation’s school-age population and the budget they proposed in their application for 
accomplishing the specific plans they proposed. In this report, we use the term early RTT states 
to refer to the 12 states selected in the first two rounds. We use the term later RTT states for the 7 


(continued) 

ways: (1) by supporting states’ efforts to increase the number and percentage of disadvantaged children enrolled in 
high-quality early learning and development programs and (2) by designing and implementing an integrated system 
of high-quality early learning and development programs and services. RTT-District grants totaling $520 million 
were made to 21 school districts in December 2012 and 2013 to support locally developed plans with four goals: (1) 
to personalize and deepen student learning, (2) to directly improve student achievement and educator effectiveness, 
(3) to close achievement gaps, and (4) to prepare every student to succeed in college and their careers. In 2014, the 
Obama administration proposed investing $300 million in a new RTT competition promoting educational equity and 
opportunity. 

3 The Executive Summary is available at http://www2.ed.gov/programs/racetothetop/executive-summary.pdf. 

4 Title I, Part A (Title I) of the Elementary and Secondary Education Act provides financial assistance to LEAs 
and schools with high numbers or high percentages of children from low-income families. Title I funds are allocated 
using formulas that are based primarily on census poverty estimates and the cost of education in each state. 
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states selected in the third round. The early and later states were judged by ED to have 
articulated comprehensive plans to improve education policies and practices and to have 
demonstrated the capacity for, and significant past progress in, implementing policies and 
practices in the program’s topic areas. The RTT applicants, their application scores from Rounds 
1 and 2, the award amounts, and the percentage of LEAs participating for each grantee are listed 


Table 1.1. Information about RTT Competition Rounds 1, 2, and 3 



Round 1 

Round 2 

Round 3 

Eligible applicants 

Open to all states, the 
District of Columbia, 
and Puerto Rico 

Open to all states other 
than Round 1 grantees 
and to the District of 
Columbia and Puerto 
Rico 

Open only to the nine 
finalists from Round 2 
that did not receive a 
grant in Round 2 

Policies and Practices 

ED asked states to 
address policies and 
practices in all six 
areas listed in the RTT 
application criteria 

ED asked states to 
address policies and 
practices in all six areas 
listed in the RTT 
application criteria 

ED only required 
states to focus on a 
portion of the policies 
and practices in their 
Round 2 applications 3 

Budget 

ED provided 
suggested budget 
ranges for each state 

States required to stay 
within the Round 1 
budget ranges provided 

ED provided 
suggested budget 
amounts for each state 

Timing of awards 

March 2010 

August 2010 

December 201 1 

Number of states 
selected 

2 

10 

7 

Award amounts 

$100 million-$500 
million 

$75 million— $700 
million 

$17 million— $43 million 


Source: U.S. Department of Education. 

Note: Alaska, North Dakota, Texas, and Vermont did not apply for RTT grants in any round. ED = 

U.S. Department of Education. 


a All seven Round 3 states focused on policies and practices in the areas of state capacity and standards 
and assessments. Six states also focused on policies and practices in the areas of data systems 
(Colorado was the exception) and teachers and principals (Arizona was the exception). One state 
(Louisiana) also focused on policies and practices in the area of school turnaround. Two states (New 
Jersey and Pennsylvania) also focused on policies and practices in the area of charter schools. 

in Table 1.2. 5 States have four years from the time of the award in which to implement their plans 
and spend the grant funds, although states may request a no-cost extension if they anticipate 
needing additional time to complete certain projects (as of November 2014, Delaware, the 
District of Columbia, Florida, Georgia, Maryland, Massachusetts, New York, North Carolina, 
Ohio, Rhode Island, and Tennessee had received approval for no-cost extensions). Note that we 
sometimes refer to states when discussing RTT states while recognizing that one early RTT state 
is the District of Columbia. 


5 Scores given to applications provide important context about ways in which RTT states and non-RTT states 
may have differed before RTT awards were announced. Additional analysis of the average scores for each group of 
states is presented in Chapter III. ED did not calculate scores in Round 3. 
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Table 1.2. Information about RTT Applicants and Grantees 


Round 


State 

of 

Award 

Round 1 
Score 

Round 2 
Score 

Award Amount 

Percentage of LEAs 
Participating 

Delaware 

1 

455 

a 

$100 million 

100 

Tennessee 

1 

444 

a 

$500 million 

100 

Georgia 

2 

434 

446 

$400 million 

14 

Florida 

2 

431 

452 

$700 million 

96 

Rhode Island 

2 

419 

451 

$75 million 

96 

Ohio 

2 

419 

441 

$400 million 

53 

North Carolina 

2 

414 

442 

$400 million 

100 

Massachusetts 

2 

411 

471 

$250 million 

70 

New York 

2 

409 

465 

$700 million 

86 

District of Columbia 

2 

402 

450 

$75 million 

63 

Hawaii 

2 

365 

462 

$75 million 

100 

Maryland 

2 

a 

450 

$250 million 

92 

Illinois 

3 

424 

427 

$43 million 

C 

Pennsylvania 

3 

420 

418 

$41 million 

c 

Kentucky 

3 

419 

412 

$17 million 

c 

Louisiana 

3 

418 

434 

$17 million 

72 

Colorado 

3 

410 

420 

$18 million 

51 

New Jersey 

3 

387 

438 

$38 million 

28 

Arizona 

3 

240 

435 

$25 million 

C 

South Carolina 

b 

423 

431 

b 

b 

Arkansas 

b 

394 

389 

b 

b 

Utah 

b 

379 

379 

b 

b 

Minnesota 

b 

375 

a 

b 

b 

Michigan 

__b 

366 

382 

b 

b 

Indiana 

__b 

356 

a 

b 

b 

Iowa 

b 

346 

383 

b 

b 

Connecticut 

b 

345 

379 

b 

b 

Wisconsin 

__b 

341 

368 

b 

b 

California 

__b 

337 

424 

b 

b 

Idaho 

__b 

331 

a 

b 

b 

Kansas 

__b 

330 

a 

b 

b 

New Mexico 

b 

325 

366 

b 

b 

Virginia 

__b 

325 

a 

b 

b 

Wyoming 

b 

319 

a 

b 

b 

Missouri 

__b 

301 

316 

b 

b 

Oklahoma 

b 

295 

392 

b 

b 

Oregon 

__b 

293 

a 

b 

b 

West Virginia 

__b 

292 

a 

b 

b 

Alabama 

__b 

291 

212 

b 

b 

New Hampshire 

__b 

271 

335 

b 

b 

Nebraska 

__b 

247 

296 

b 

b 
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Table 1.2 (continued) 


State 

Round 

of 

Award 

Round 1 
Score 

Round 2 
Score 

Award Amount 

Percentage of LEAs 
Participating 

South Dakota 

__b 

136 

a 

b 

b 

Maine 

__b 

__a 

283 

b 

b 

Mississippi 

__b 

__a 

263 

b 

b 

Montana 

b 

__a 

238 

b 

b 

Nevada 

__b 

a 

381 

b 

b 

Washington 

b 

__a 

291 

b 

b 


Source: U.S. Department of Education. 

Note: States are listed within rounds by their Round 1 score. Alaska, North Dakota, Texas, and 

Vermont did not apply for RTT grants in any round. The U.S. Department of Education did not 
calculate scores in Round 3. 

a Delaware and Tennessee do not have Round 2 scores because they received grants in Round 1. 
Maryland, Maine, Mississippi, Montana, Nevada, and Washington do not have Round 1 scores because 
they did not apply for an RTT grant in Round 1. Minnesota, Indiana, Idaho, Kansas, Virginia, Wyoming, 
Oregon, West Virginia, and South Dakota do not have a Round 2 score because they did not apply for an 
RTT grant in Round 2. 

b These states did not receive a grant in any round. 

c These states did not report the percentage of LEAs participating or provide a list of participating LEAs in 
their RTT Round 3 applications. 

LEA = local education agency. 

B. Prior Research on RTT 

Given that it is a relatively new program, prior research on RTT implementation is limited. 
Several studies have examined RTT implementation progress in RTT states. These studies found 
that RTT states made progress in implementing RTT policies and practices, though many states 
encountered challenges that caused delays (Boser 2012; Government Accountability Office 
2011; Government Accountability Office 2015; Miller and Hanna 2014; 
http://www2.ed.gov/programs/racetothetop/performance.html). 6 One study that evaluated early 
RTT states’ first- year efforts to meet the goals outlined in their applications rated 10 out of the 
12 early RTT states as “meeting expectations” (Boser 2012). A follow-up study found that all 
RTT states adopted college- and career-ready standards, nearly all RTT states developed data 
systems to track students from preschool into postsecondary education, and four RTT states were 
at or near full implementation of new educator evaluation systems aligned with RTT 
requirements (Miller and Hanna 2014). ED determined that most states were making satisfactory 
progress in their first and second years of implementation and placed the few exceptions on high- 
risk status, a designation that can result in loss of grant funding. Although these studies 
examined RTT implementation progress, they did not examine as many RTT-promoted policies 
and practices as we examine in this report, they did not examine the extent to which non-RTT 
states might have adopted these policies or practices, nor did they synthesize them into the six 
areas described in the RTT application criteria. 


6 U.S. Department of Education “RTT Annual Performance Report" (http://www2.ed.gov/programs/ 
racetothetop/performance.html) provides RTT annual performance reports for each state. 
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RTT promotes a broad set of policies and practices, and several studies have shown that 
RTT states have faced more difficulties implementing some types of policies and practices than 
others. In particular, many states are experiencing difficulties implementing teacher and 
principalevaluation systems and developing data systems (Boser 2012; Weiss 2013; 
http://www2.ed.gov/programs/racetothetop/performance.html). Common challenges that states 
have encountered during RTT implementation include procurement delays, staff turnover, and 
legal challenges and contract disputes over the teacher and principal evaluation systems. 

Most research on the implementation of policies and practices promoted by RTT focuses 
only on RTT states, but a few studies (Government Accountability Office 2011; Howell 2015; 
National Council on Teacher Quality 2011) that include non-RTT states suggest that RTT states 
were more likely to adopt these types of policies and practices than other states. Officials from 
states that applied for and received RTT grants reported that the RTT funds would accelerate 
their adoption of policies and practices; officials from states that applied for but did not receive 
RTT grants reported that they still planned to implement some of the proposed policies and 
practices, though more slowly than if they had received an RTT grant (Government 
Accountability Office 2011). One study of states’ teacher evaluation policies after the first year 
of RTT implementation found that early RTT states were more likely to have adopted legislation 
or regulations that gave student achievement a significant and meaningful role in teacher 
evaluations than later RTT states or non-RTT states (National Council on Teacher Quality 2011). 
Finally, a recent study found that while both RTT and non-RTT states enacted a subset of RTT- 
promoted policies at a higher rate after the onset of RTT than previously, policy adoption rates 
increased more for RTT states than non-RTT states (Howell 2015). 

A large literature examines the process of state policy adoption more broadly. Because the 
goal of this report is to provide descriptive information about the extent to which states used the 
policies and practices promoted by RTT, rather than to explain why some states used these 
policies and practices and other states did not, it is beyond the scope of this report to summarize 
this literature in detail. It includes studies that use statistical modeling to predict whether certain 
states will adopt policies or not, based on state characteristics and the history of policy adoption. 
To name two examples, Berry and Berry (1990) examined state adoption of lotteries and 
Mintrom (2000) examined state adoption of school choice policies. 

This report focuses on the usage of RTT-promoted policies and practices, and sets the table 
for a future report that will examine whether receipt of an RTT grant is related to improvement 
in student outcomes. Because this report does not examine the effectiveness of RTT, it is beyond 
the scope of this report to discuss the literature on RTT effectiveness in detail; however, we 
provide a brief summary of it in the next paragraph to set the stage for the future report. 

To our knowledge, no studies to date have examined the impact of RTT on student 
outcomes or other measures of educational improvement. A vast literature does examine the 
effectiveness of the types of policies and practices promoted by RTT. Overall, this literature 
provides no conclusive evidence on whether RTT-promoted policies and practices improve 
student outcomes. In two of the six areas described in the RTT application criteria (standards and 
assessments, school turnaround), to our knowledge, no experimental studies examined the 
relationship between RTT-promoted policies and practices and student outcomes, and non- 
experimental studies found mixed results (some studies found that RTT-promoted policies and 
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practices were associated with improved student outcomes, while other studies found no 
relationship between these policies and practices and student outcomes). 7,8 In three areas (data 
systems, teacher and principal certification and evaluation, charter schools), the evidence from 
both experimental and non-experimental studies was mixed. 7 8 9 In the area of state capacity, to our 
knowledge, no studies examined the relationship between the types of policies and practices 
promoted by RTT and student outcomes. 

C. Evaluation Focus 

As noted in section B above, there is no conclusive evidence on whether the types of 
policies and practices promoted by RTT improve student outcomes, and no studies have 
examined the relationship between the RTT program and student outcomes. In addition, few 
studies on the implementation of RTT-promoted policies and practices examine a comprehensive 
set of these policies and practices spanning all the areas described in the RTT application, and 
few examine whether the policies and practices used by RTT states differ from those used by 
non-RTT states. This evaluation seeks to address these gaps in the existing literature. It is 
designed to examine the relationship between receipt of an RTT grant and student achievement 
(a topic that will be addressed in a future report), whether RTT grant recipients report using the 
policies and practices promoted by the program, and if they are doing so to a different extent 
than states that have not received such grants. This volume of the report focuses on the following 
research questions: 

• Are RTT states using the educational policies and practices promoted by RTT, and 
how does that compare to the usage of those policies and practices by non-RTT 
states? 

• Does usage of these policies and practices include a focus on ELLs, and does that 
focus on ELLs vary between RTT and non-RTT states? 

• Does usage of these ELL-focused educational policies and practices vary by the 
percentage of students who are ELLs or the ELL/non-ELL student achievement gap? 

To address these research questions, we conducted interviews with state-level administrative 
staff in 49 states and the District of Columbia. 


7 Experimental studies are those that examine treatment and comparison groups from a randomized controlled 
trial. The results from experimental studies (if such studies are conducted appropriately) can be used to make causal 
statements about the effect of a practice, policy, program, or intervention on an outcome of interest. The results from 
non-experimental studies cannot be used to make such causal statements because these studies cannot rule out the 
possibility that other differences between the groups — besides the intervention itself — caused any observed 
differences in outcomes. 

8 To cite a few examples of these non-experimental studies: Ballou and Springer 2008; Bandeira de Mello 
2011; Bifulco et al. 2003; Borman et al. 2003; Booker et al. 2009; Dee 2012; Dee and Jacob 2011; de la Torre and 
Gwynne 2009; Figlio et al. 2009; Fryer 2014; Fuller et al. 2007; Flanushek and Raymond 2005; Lee 2006; Player 
and Katz 2013; and Strunk et al. 2012. 

9 To cite a few examples: Abdulkadiroglu et al. 2011; Allen et al. 2011; Carlson et al. 2011; Clark et al. 2013; 
Constantine et al. 2009; Decker et al. 2004; Dobbie and Fryer 2011; Fryer et al. 2012; Garet et al. 2008; Garet et al. 
2010; Glazerman et al. 2006; Glazerman et al. 2013; Gleason et al. 2010; Flenderson et al. 2007; Flenderson et al. 
2008; Floxby et al. 2009; Floxby and Rockoff 2005; May and Robinson 2007; Quint et al. 2008; Slavin et al. 2011; 
Tuttle et al. 2013; Zimmer and Buddin 2006; and Zimmer et al. 2012. 
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D. Looking Ahead 

In Chapter II, we describe the study sample, design, and data collected to address these 
research questions. In Chapter III, we provide baseline information on the RTT sample. In 
Chapter IV, we present findings on the extent to which RTT states reported using the policies 
and practices promoted by RTT in spring 2012 and how that compares to the usage of those 
policies and practices by non-RTT states. In Chapter V, we present findings on the extent to 
which RTT and non-RTT states reported focusing on ELLs in their usage of the policies and 
practices promoted by RTT, and how that varies by the percentage of students who are ELLs and 
the ELL/non-ELL achievement gap. In Chapter VI, we discuss the findings from this volume. In 
Appendices A, B, C, and D, we provide additional results, including responses to individual 
interview questions. 
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II. STUDY SAMPLE, DATA COLLECTION, AND ANALYSIS 

In this chapter, we describe the study sample, the data collected, and the method of 
analyzing the data for the evaluation of RTT. 

A. Study Sample and Data Collection 

The sample for the evaluation of RTT includes 49 states and the District of Columbia. The 
data examined in this volume come from structured telephone interviews with representatives 
from the state education agencies in 49 states and the District of Columbia, conducted in spring 
2012, for a response rate of 98 percent (Texas did not participate in the interviews). The 
interviews took place roughly 25 months after Round 1 grants were received, 20 months after 
Round 2 grants were received, and 4 months after Round 3 grants were received. These 
interviews collected infonnation about educational policies, practices, and supports related to the 
six primary RTT topic areas. The RTT objectives in each area and the policies and practices 
within each area for which we had state interview data are detailed in Table II. 1. 10,11 The state 
interview included questions addressing twelve ELL-focused policies and practices aligned with 
RTT objectives (Table II.2). Interviews were organized into modules (one for each of the six 
areas), and we typically interviewed different respondents from each state for each module. 
Respondents were the state administrators most knowledgeable about each area. Each question 
received only one response per state. To facilitate comparisons between RTT states and non-RTT 
states, we asked all states the same set of questions. To limit the length of the interview, we 
sought input from IES and the RTT program office on which questions were of greatest interest 
to them. We conducted pilot tests of the instrument and provided training to the data collection 
team to ensure the uniformity and consistency of the data collected. 

To provide baseline and other contextual infonnation for the sample used in this report, we 
drew on two other publicly available data sources: National Assessment of Educational Progress 
(NAEP) scores and the Common Core of Data. From NAEP, a nationally representative 
assessment of U.S. students, we obtained state-level average scores in math and reading for 4th 
and 8th grades from 2007. From the Common Core of Data, which includes annual data about 
each public school, LEA, and state in the country, we obtained 2007-2008 state-level 
characteristics such as the percentage of students in each race or ethnicity category, the 
percentage of students eligible for free or reduced-price lunch (an indicator of economic need), 
the percentage of schools eligible for Title I (a program that provides financial assistance to 
LEAs and schools with high numbers or high percentages of children from low-income families), 
the percentage of schools at each level (elementary, middle, high), and the average number of 
full-time staff. 


10 The state interview protocol is available at http://www.mathematica-mpr.com/publications/PDFs/Spring_ 
20 1 2_State_Interview_Protocol.pdf. 

11 The interview questions specified particular time periods of interest. For example, some questions asked, 
“Currently, are you...?” and others asked “In the 2007-2008 school year, did you...?” Therefore, the results in this 
report represent a point-in-time measure of the policies and practices used by states. Throughout the report, we use 
phrases such as “In spring 2012, states reported using policies and practices” to reflect that these are point-in-time 
measures. Because the usage of policies and practices is a dynamic process, it is possible that some policies and 
practices used by states at the time of the interview were no longer in use after the interview. For example, some 
states that had adopted Common Core standards at the time of the interview may have repealed them afterwards. 
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Table 11.1. RTT Objectives and Policies and Practices Addressed by State Interview Questions, by Topic Area 


Topic Area 

RTT Objectives 

Policies and Practices Addressed by State Interview Questions 

Improving state 
capacity to support 
school improvement 
efforts 

Articulating the state’s 
education reform agenda and 
local education agencies’ 
participation in it 6 

• Have reform plan in place 

• Prioritize college and career-ready standards or assessments 

• Prioritize development of state longitudinal data system (SLDS) or 
data use 

• Prioritize teacher and principal evaluation and effectiveness 

• Prioritize school turnaround 


Building strong statewide 
capacity to implement, scale 
up, and sustain the proposed 
plans 

• Support implementation of reform plan 

• Work with districts to replicate best practices statewide 

• Provide effective and efficient grant oversight 

• Involve teachers, unions, or administrators to define priorities 

• Involve other stakeholders to define priorities 


Demonstrating significant 
progress in raising 
achievement and closing gaps 

No items in state interview aligned with this objective 11 

Adopting standards 
and assessments that 
prepare students to 
succeed in college 
and the workplace 

Developing and adopting 
common standards 

Developing and implementing 
common, high-quality 
assessments 

• Adopt Common Core State Standards (CCSS) 

• Participate in CCSS Consortium 

• Fully implement assessments developed by consortium 


Supporting the transition to 
enhanced standards and high- 
quality assessments 

• Provide funds for new staff or new technology to implement CCSS 

• Provide new technology for CCSS assessments 

• Require new curricula or provide materials to implement CCSS 

• Require or support development or implementation of interim 
assessments 

• Change high school exit exam, credit, or college entrance 
requirements 

• Provide support or professional development to teachers or 
principals to implement CCSS 

• Support CCSS implementation for English language learners 
(ELLs) 

Building state data 
systems that measure 
student growth and 
inform instruction 

Fully implementing a statewide 
longitudinal data system 

• Have SLDS 

• SLDS has program participation data 

• SLDS linked to early childhood data system 

• SLDS linked to higher education data system 


Accessing state data and 
using it to inform key 
stakeholders 

• Allow SLDS to be accessed by stakeholders 

• Use SLDS to support decision makers 


Using data to improve 
instruction 

• Require districts to implement data system 

• Provide supports to districts for using data 

Recruiting, 
developing, 
rewarding, and 
retaining effective 
teachers and 
principals 

Providing high-quality 
pathways to certification for 
aspiring teachers and 
principals 

• Authorize qualified alternative-route program operators 6 

• Increase alternative-route program selectivity 

• Increase time in school-based learning experiences 

• Allow alternative-route programs to award same certification as 
traditional-route programs' 1 

• Have alternative-route programs currently operating 

• Have process to identify teacher and principal shortage areas 

• Take steps to address teacher and principal shortage areas 


Improving teacher and 
principal effectiveness based 
on performance 

• Require student achievement growth 

• Require multiple evaluation measures 

• Specify required minimum number of rating levels 


Conduct annual teacher evaluations 6 

Require that evaluation results be used for professional 

development 

Require that evaluation results be used for compensation 
decisions 

Require that evaluation results be used for career advancement 
decisions 

Require that evaluation results be used for dismissal decisions 
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Table 11.1 (continued) 

Topic Area RTT Objectives Policies and Practices Addressed by State Interview Questions 

Ensuring equitable distribution • Require use of strategies for more equitable distribution of 
of effective teachers and effective teachers and principals 

principals . Conduct analyses of shifts in distribution of effective teachers and 

principals 

Improving the effectiveness of • Use effectiveness ratings to assess certification programs 

teacher and principal • Publicly report certification program effectiveness 

preparation programs . Support effective certification programs 

Providing effective support to No items in state interview aligned with this objective 11 
teachers and principals 

Turning around low- Authority to intervene in the • Have authority to take over failing schools 

performing schools lowest-achieving schools and 
local education agencies 

• Provide training in analyzing data to improve instruction 

• Help align curricula to state standards 

• Provide training on school improvement plans, effective curricula, 
instructional strategies, or intervention models 

• Provide training on strategies for ELLs 

• Provide technical assistance on improving professional 
development 

• Provide operational flexibility for staffing and budgeting 

• Provide technical assistance on recruiting and retaining teachers 
or financial incentives 

• Have teacher tenure rules that address placement in or removal 
from lowest-achieving schools 

• Have administrative structures for supporting turnaround efforts, 

monitoring, or providing support 

Encouraging Eliminating restrictions on • Have no restrictions on new charter schools or charter enrollment 

conditions in which charter school creation and 

charter schools can enrollment 

succeed 

Refining authorization and • Prioritize schools that address needs of ELLs 

monitoring processes . Monitor academic performance of charter schools 

» Monitor nonacademic performance of charter schools 

Source: RTT application; interviews with state administrators in spring 2012. 

a Regular school districts are the most common type of local education agency. 

b The number of questions included in the state interview was purposefully limited to reduce the time it took to complete the 
interview. We initially developed the interview questions based on an examination of the RTT application criteria. To ensure that the 
interview was of a reasonable length, we then pared down the initial list of questions through a deliberative process with the Institute 
of Education Sciences and the RTT Program Office, to assess their priorities for the types of questions to include. The interview did 
not include any questions about this objective. 

c Alternative-route programs are those that offer an alternative route to certification. 
d Traditional-route programs are those that offer a traditional route to certification. 
e The state interview did not ask about this policy or practice for principals. 

Table 11.2. RTT Objectives and the ELL-Focused Policies and Practices Aligned with Those Objectives That Were 
Addressed by State Interview Questions, by Topic Area 

ELL-Focused Policies and Practices Addressed by State Interview 
Topic Area RTT Objectives Questions 

Improving state Articulating the state’s • Prioritize support to ELLs 

capacity to support education reform agenda and 

school improvement local education agencies’ 

efforts participation in it a 

Building strong statewide . Provide targeted support to ELLs 

capacity to implement, scale . Implement organizational or administrative changes to improve 

up, and sustain the proposed capacity to support ELLs 

plans 


Turning around the lowest- 
achieving schools 
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Table 11.2 (continued) 

ELL-Focused Policies and Practices Addressed by State Interview 
Topic Area RTT Objectives Questions 

Demonstrating significant No items in state interview aligned with this objective for ELLs b 

progress in raising 
achievement and closing gaps 

Developing and adopting No items in state interview aligned with this objective for ELLs b 

common standards 

Developing and implementing No items in state interview aligned with this objective for ELLs b 
common, high-quality 
assessments 

Supporting the transition to • Support Common Core State Standards implementation for ELLs 

enhanced standards and high- 
quality assessments 

Building state data Fully implementing a statewide 
systems that measure longitudinal data system 
student growth and 

inform instruction Accessing state data and 

using it to inform key 
stakeholders 

Using data to improve • Provide supports to districts for using ELL-related data 

instruction 

Providing high-quality No items in state interview aligned with this objective for ELLs b 

pathways to certification for 
aspiring teachers and 
principals 

Improving teacher and No items in state interview aligned with this objective for ELLs b 

principal effectiveness based 
on performance 

Ensuring equitable distribution No items in state interview aligned with this objective for ELLs b 
of effective teachers and 
principals 

Improving the effectiveness of No items in state interview aligned with this objective for ELLs b 
teacher and principal 
preparation programs 

Providing effective support to No items in state interview aligned with this objective for ELLs b 
teachers and principals 

Turning around low- Authority to intervene in the No items in state interview aligned with this objective for ELLs b 

performing schools lowest-achieving schools and 

local education agencies 

Turning around the lowest- • Offer financial incentives for teachers with ELL expertise 

achieving schools • Provide training on strategies for ELLs 

• Have state-level staff to support turnaround schools and districts in 
working with ELLs 

Encouraging Eliminating restrictions on No items in state interview aligned with this objective for ELLs b 

conditions in which charter school creation and 

charter schools can enrollment 

succeed 

Refining authorization and . Prioritize schools that address needs of ELLs 

monitoring processes . Monitor charter school performance based on the student 

populations (such as ELLs) served 

Source: RTT application; interviews with state administrators in spring 2012. 

ELL = English language learner. 

a Regular school districts are the most common type of local education agency. 

b The number of questions included in the state interview was purposefully limited to reduce the time it took to complete the 
interview. We initially developed the interview questions based on an examination of the RTT application criteria. To ensure that the 
interview was of a reasonable length, we then pared down the initial list of questions through a deliberative process with the Institute 
of Education Sciences and the RTT Program Office, to assess their priorities for the types of questions to include. The interview did 
not include any questions about this objective. 


Recruiting, 
developing, 
rewarding, and 
retaining effective 
teachers and 
principals 


• State longitudinal data system has program participation data 
about ELLs 

• Use state longitudinal data system to support decision makers in 
improvement efforts for ELLs 


Adopting standards 
and assessments that 
prepare students to 
succeed in college 
and the workplace 
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B. Analysis Methods 

In this section, we describe the methods we used to compare the policies and practices 
reported by RTT states and non-RTT states. The purpose of these comparisons is not to 
detennine whether receipt of an RTT grant caused states receiving those funds to use particular 
policies and practices, but simply to detennine whether states receiving an RTT grant used the 
policies and practices promoted by RTT, and how that compares to the usage of those policies 
and practices by states that did not receive an RTT grant. In interpreting the results, please note 
the following caveats: (1) the findings are based on self-reported usage of policies and practices, 

(2) our study instruments did not address every policy and practice listed in the RTT application, 

(3) the application wording left it up to the states to decide many of the details about how to 
implement particular policies and practices, and (4) we did not collect infonnation about the 
quality or fidelity with which the policies and practices were implemented. 12 While the study 
team took several steps to try to ensure the quality and accuracy of the data 13 , the data provided 
by states were self-reported and not independently verified by the study team. Thus, readers 
should use caution when interpreting the results. 

We first describe how we formed the groups that are the basis for the comparisons presented 
in this volume of the report. We then describe how we summarized the large number of findings 
and how we analyzed the extent to which states focused on ELLs in their usage of policies and 
practices promoted by RTT. 


12 Here we provide several potential explanations for how these limitations might affect the data. Our data do 
not allow us to determine whether any of these possible explanations are correct, but we offer them here as starting 
points for thinking about how the results might be affected by the limitations. Self-reported levels of policy and 
practice usage might be overestimated (relative to actual usage) if states provided socially desirable responses. This 
would likely lead to overestimated levels for all states. The study team took several steps to ensure that states 
provided accurate responses, including telling states that the interview was not an audit and that we would report 
aggregated responses across states, rather than singling out any individual state. The fact that self-reported levels of 
policy and practice usage (as presented in Chapter IV) are not all 100 percent, and in many cases are much lower 
than 100 percent, suggests that the issue of providing socially desirable responses is not a rampant problem in our 
data. However, it is possible that RTT states might be more likely than non-RTT states to provide socially desirable 
responses, given that they received RTT grants to implement the policies and practices we examined. Therefore, the 
results for RTT states might be more inflated than the results for non-RTT states, so readers should use caution 
when interpreting the results. Regarding the fact that our study instruments did not address every policy and practice 
listed in the RTT application criteria, this is unlikely to have a large effect on the overall results because there were 
very few policies and practices not addressed by our interview questions (Table II. 1 shows that only 2 out of 18 RTT 
objectives were not addressed by interview questions). Regarding the quality or fidelity with which the policies and 
practices were implemented, and the fact that the application wording left it up to the states to decide many of the 
details about how to implement particular policies and practices, our data might overestimate usage levels if states 
tended to report that they used a policy or practice if they had at least begun to use it, but hadn’t necessarily 
implemented it fully. This would lead to overestimated levels for all states, rather than affecting the differences 
between RTT and non-RTT states. In addition, as noted above, the fact that many of the self-reported levels of 
policy and practice usage (as presented in Chapter IV) are well below 100 percent suggests that these levels might 
not be overestimated by a substantial amount. 

13 The study team conducted pilot tests of the interview instrument and provided training to the data collection 
team to ensure the uniformity and consistency of the data collected. The interview consisted of mostly closed-ended 
questions — that is, questions with yes or no responses or with a set of specific response categories from which to 
choose. The study team carefully reviewed all responses for completeness and consistency, and followed up with 
respondents about missing and inconsistent responses to ensure their accuracy. 
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1. RTT Comparisons 

We conducted two sets of RTT comparisons: (1) Round 1 and 2 RTT states with non-RTT 
states and (2) Round 3 RTT states with non-RTT states. 14 We distinguish between Rounds 1 or 2 
and Round 3 to account for differences in the grants’ timing, funding levels, and scope for these 
groups of states (described in Chapter I). We combined Round 1 and 2 RTT states for several 
reasons: the funding amounts and scope of the grants in those rounds were similar, only two 
states received Round 1 RTT grants, and the first two rounds of grants were awarded within 
roughly five months of each other. 

The rationale for comparing Round 3 RTT states with non-RTT states is to provide 
descriptive infonnation about the extent to which these states were using RTT-promoted policies 
and practices in spring 2012. The purpose of including these comparisons in this report is not to 
evaluate the impact of RTT grant receipt on usage of RTT-promoted policies and practices, but 
rather, to provide an early look at the extent to which later RTT states were using RTT-promoted 
policies and practices, and how that compared to usage of those policies and practices by non- 
RTT states. The results provide important context for interpreting future outcomes for these two 
groups of states. 

When the data for this report were collected in spring 2012, Round 1 and 2 states had been 
implementing their grants substantially longer than Round 3 states (approximately 20 to 25 
months for Round 1 and 2 states versus approximately 4 months for Round 3 states). In addition, 
less funding was available for Round 3 states, so they were asked to focus on a subset of areas. 
In particular, all seven Round 3 states focused on the areas of state capacity and standards and 
assessments, six focused on data systems and teacher and principal certification and evaluation, 
two focused on charter schools, and one focused on school turnaround. For these reasons, one 
might expect different progress from Round 3 states than from Round 1 and 2 states. 

2, Summarizing Findings from the State Interviews 

Given the large number of questions in the interviews, it is difficult to discern broad patterns 
or form overall conclusions by only separately examining responses to individual questions. 
Therefore, we analyzed data from the interviews using methods designed to provide information 
about broad patterns observed in the data. Readers interested in the responses to specific 
interview questions can refer to Appendix B. The process of summarizing findings involved 
several steps: 

1. Selecting subtopics. For each of the six broad topic areas in the RTT application 
criteria (listed in Chapter I and Table II. 1 above), we selected subtopics of interest 
using the RTT application criteria as a guide. For example, the standards and 
assessments section of the application criteria included subsections on (1) developing 
and adopting common standards; (2) developing and implementing common, high- 
quality assessments; and (3) supporting the transition to enhanced standards and high- 
quality assessments. We used each of those three subsections as subtopics in our 
analysis. 


14 We did not compare Round 1 and Round 2 states to Round 3 states because this report is focused on 
comparing states that received RTT grants to those that did not. 
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2. Selecting questions aligned with the RTT application criteria. For each subtopic, 
we then used a systematic approach to select interview questions that aligned with the 
policies and practices that RTT sought to affect in that area. First, a Mathematica 
researcher selected questions corresponding to each policy or practice based on 
whether he or she determined them to be aligned with the RTT application materials. 
Another Mathematica researcher then reviewed those decisions to ensure that he or 
she agreed with each one (the agreement rate was 97 percent). 15 A third Mathematica 
researcher resolved any differences of opinion between the first two researchers. We 
determined the topic area and subtopic into which each interview question fell based 
on the section of the RTT application criteria with which it aligned. By design, the 
modules in the state interviews directly map to sections of the application criteria. We 
did not use an interview question for more than one subtopic because that would have 
resulted in the question being overweighted in the overall topic area. When a question 
could potentially be used for more than one subtopic, we assigned it to the subtopic 
(and corresponding section of the application criteria) with which it was best aligned. 
The interview questions addressed all 6 topic areas and all but 2 of 22 subtopics from 
the application criteria. 

3. Constructing policy and practice variables from interview questions. For each 
policy or practice in the RTT application criteria for which we identified one or more 
relevant questions, we constructed a variable ranging from zero to one using those 
questions. A value of one indicates that the state responded “yes” to all the questions 
aligned with that policy or practice in the application criteria, a value of zero indicates 
that the state responded “yes” to none of the questions aligned with that policy or 
practice, and a value between those two limits indicates that the state responded “yes” 
to some of the questions aligned with that policy or practice. 16 This process resulted 


15 It was important to have a second researcher review these decisions to ensure that the interview questions 
selected for each subtopic were aligned with the RTT application criteria. Measurement of the extent to which the 
first and second researchers agreed on which questions were aligned with the RTT application criteria is called 
“inter-rater reliability” in statistics. Inter-rater reliability is traditionally measured using the percent agreement rate, 
calculated as the number of questions for which the first and second researchers agreed on whether or not the 
question was aligned with the RTT application criteria, divided by the total number of interview questions (Gwet 
2014 ). 

16 Many questions were originally structured with two response options, with a response of “yes” (recoded to a 
value of one) indicating that the state reported having the policy or practice in place and a response of “no” (recoded 
to a value of zero) indicating that the state did not report having the policy or practice in place. In some cases, 
however, it was necessary to combine multiple questions to determine whether or not a state reported fully using a 
particular policy or practice. For example, one criterion in the application is that teachers be evaluated annually, but 
the interview asked separate questions about the frequency of evaluation for several different types of teachers. In 
this case, we required that a state respond “yes” to all these questions to receive a value of one for the variable 
corresponding to this policy or practice. In other cases, any one of several questions could be counted as partial 
evidence of a state’s usage of a policy or practice aligned with the RTT application criteria. For example, the 
application criteria specified that the state use support from a broad group of stakeholders to implement its education 
policy plans. The interview included eight separate questions about whether each of eight different types of 
stakeholders (such as local school boards and community organizations) were involved in defining the state’s 
education policy plans. In these cases, a state received a fraction of a point (in this example, a value of one-eighth) 
for each “yes” response. This approach helped to ensure that we did not overweight some interview questions 
relative to how they were represented in the application criteria. 
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in the following numbers of policies and practices for each topic area: 10 policies and 
practices for state capacity, 10 policies and practices for standards and assessments, 8 
policies and practices for data systems, 39 policies and practices for teacher and 
principal certification and evaluation, 10 policies and practices for school turnaround, 
and 4 policies and practices for charter schools. The emphasis (that is, the number of 
policies and practices) given to each area and the specificity of the policies and 
practices within each area directly relates to how much emphasis and specificity was 
provided in the RTT application criteria for these areas. 17 

4. Summing the policies and practices for each state. To detennine each state’s 
progress in using policies and practices aligned with RTT, we summed the variables 
created in step 3. This sum was calculated separately for each subtopic. We then 
summed across subtopics to create a sum for each topic area. 18 If a particular state 
was missing values for a particular policy or practice, we took the mean of the non- 
missing policies and practices and multiplied it by the total number of policies and 
practices for the overall area. For example, for the state capacity area, which has 10 
policies and practices, if a state had data available for 5 policies and practices, and 
reported using 2 of them, the number of the state’s reported used policies and 
practices would be equal to (2/5)* 10. Across all states and all subtopics, the average 
percentage of policies and practices that were missing was 1 .7 percent. 19 

5. Averaging the number of policies and practices across states. For each group of 
states (early RTT, later RTT, non-RTT), we averaged the numbers calculated in step 
4. We calculated this average number of policies and practices reported for the three 
groups of states separately for each topic area and subtopic. 


17 For example, the teacher and principal certification and evaluation area was more heavily emphasized in the 
RTT application criteria than any other area. This is demonstrated by the number of pages devoted to this area in the 
RTT application (11.5 pages, versus 8, 3.5, 3, 2.5, and 2 pages for state capacity, standards and assessments, data 
systems, school turnaround, and charter schools), the percentage of points received by this section of the application 
(28 percent, versus 15, 14, 9, 10, and 8 percent for the other areas), and the number of criteria in this section of the 
application (9 criteria, versus 7, 4, 3, 3, and 1 for the other areas). Correspondingly, our state interview was 
purposefully designed to contain more questions aligned with the teacher and principal certification and evaluation 
section of the application than with the other sections. 

18 Thus, one or more interview cjuestions were used to create a variable for each policy or practice, one or more 
policies or practices formed a subtopic, and one or more subtopics formed a topic area (or area, for short). 

19 To assess how much the way we coded missing data might have affected our results, we conducted a 
bounding exercise in which we re-calculated the results twice: once setting all missing responses to “no” (that is, 
assuming all missing responses indicated that the policy or practice was not used) and once setting all missing 
responses to “yes” (that is, assuming all missing responses indicated that the policy or practice was used). The 
results were largely unchanged. The magnitude of differences between RTT and non-RTT states with respect to the 
number of RTT-promoted policies and practices used were very similar to the magnitudes reported in Chapter IV. In 
addition, across the 24 statistical significance tests conducted as part of this bounding exercise (two for each of the 
six topic areas when comparing early RTT to non-RTT states, and two for each of the six topic areas when 
comparing later RTT to non-RTT states), only two results differed from what is shown in Chapter IV: (1) when 
setting all missing responses to “yes,” the difference between early RTT and non-RTT states with respect to the 
number of RTT-promoted policies and practices used in the area of charter schools was no longer statistically 
significant; and (2) when setting all missing responses to “yes,” the difference between later RTT and non-RTT 
states with respect to the number of RTT-promoted policies and practices used in the area of teacher and principal 
certification and evaluation was no longer statistically significant. 
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6. Testing differences between groups of states. We conducted statistical tests to 
assess whether the average number of policies and practices reported differed 
between the early RTT and non-RTT states. 20 We also tested whether the average 
number differed between the later RTT and non-RTT states. Because the goal of this 
analysis was to provide descriptive information about the actual levels of policies and 
practices used by RTT and non-RTT states in spring 2012, the results are reported as 
raw (that is, unadjusted) means; they are not regression-adjusted to account for any 
pre-existing differences between RTT and non-RTT states. Throughout this report, 
we focused on the statistical significance of differences between RTT and non-RTT 
states (rather than the magnitude of differences) to ensure that consistent, objective, 
and transparent criteria were used for reporting findings. One caveat with this 
approach is that some statistically significant differences might not be substantively 
important; we indicated places in the report where this might be the case. 

This method of summarizing findings is one way to analyze broad patterns observed in the 
data, and compare levels of usage of policies and practices across different groups of states. If 
variables had been constructed differently (for example, if multiple questions that addressed the 
same policy or practice had not been combined into a single variable, but had each been included 
in the analysis as separate variables), the results might change. Therefore, it is important to keep 
these methods in mind when interpreting the results, along with the caveats mentioned above 
about the interview instrument and the wording of the RTT application criteria. 

In addition to examining the differences in policies and practices for all six areas in spring 
2012, for three areas (teacher and principal certification and evaluation, school turnaround, 
charter schools) we examined whether states differed with regard to the policies and practices 
that they reported having in place during the 2007-2008 school year. This was possible because 
the interviews collected data not only on policies and practices in place during the 2011-2012 
school year but also baseline data about selected policies and practices in these areas in place 
during the year before the announcement and implementation of RTT (2007-2008). 21 For the 


20 Although we have the entire population of states in our analysis (as opposed to a sample of states), 
hypothesis testing is still a useful way to account for the possibility that differences in outcomes between the two 
groups are due to randomness rather than due to a “true,” systematic difference. Although there is no source of 
randomness inherent in our study design (such as random sampling or random assignment), there may be other 
reasons for differences in outcomes between the two groups that are functionally random (that is, orthogonal to RTT 
status and other observable covariates). For this analysis, we used a permutation test, which is the non-parametric 
counterpart to a t-test. The statistical power of this test differed by topic area and subtopic because it depended on 
several factors, including the number of interview questions aligned with the RTT application criteria, the number of 
variables constructed from those questions, and the degree to which the variables were correlated with each other. 

21 Interview responses to questions pertaining to earlier school years (such as 2007-2008) might suffer from 
recall error. That is, responses about earlier school years might differ from those about the current year simply 
because of the respondent’s inability to recall accurately the earlier time period rather than because of actual changes 
from year to year. This error could be systematic, resulting in bias, or random, resulting in less precise estimates. To 
address this concern, we sent our interview protocols to each respondent prior to the interview. In addition to 
drawing attention to items that required respondents to look up information prior to the interview, we directed their 
attention to questions that covered the 2007-2008 school year to make sure that they were prepared to answer those 
questions during the interview. For example, they may have needed to obtain the information about that period from 
a colleague or ask that a colleague with more knowledge of that time period also participate in the interview. We 
pilot tested this process to assess the extent to which respondents were able to report information on the 2007-2008 
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other three areas, it was not possible to examine baseline data because the interview did not 
include any questions about the 2007-2008 school year that were aligned with the RTT 
application criteria for those areas. 22 We focused the baseline data collection on the 2007-2008 
school year because after the American Recovery and Reinvestment Act was announced in 
February 2009, states may have started taking steps to increase the competitiveness of their RTT 
applications in the focus areas described in the law. While the study team took several steps to 
try to improve recall accuracy (described above), data provided by states were self-reported and 
not independently verified by the study team. Thus, readers should exercise caution when 
interpreting data from 2007-2008. 

The policies and practices for which we could measure usage during the 2007-2008 school 
year are a subset of those for which we could measure usage in the 2011-2012 school year. In 
Chapter IV, we list the policies and practices for which we could measure usage in 2011-2012 
and indicate those for which we could also measure usage in 2007-2008. It is important to keep 
this in mind when interpreting the baseline results. For example, finding no statistically 
significant difference at baseline between early RTT and non-RTT states for a particular area 
means that baseline differences did not exist for some of the policies and practices in that area, 
but we do not know whether that was the case for all of the policies and practices in that area 
because we did not measure usage of all policies and practices at baseline. 

Finally, for those three areas for which we had baseline data, we examined whether usage of 
policies and practices aligned with the RTT application criteria changed over time. In Chapter 
IV, we present figures showing the change over time in the average number of policies and 
practices in place for each group of states (early RTT, later RTT, non-RTT) in these three areas. 
This analysis includes only the subset of policies and practices for which we measured usage in 
both 2007-2008 and 2011-2012. Specifically, we averaged the number of those policies and 
practices across each group of states (early RTT, later RTT, non-RTT), separately for each 
school year. We then subtracted the mean number of policies and practices for 2007-2008 from 
the mean number for 2011-2012, and tested whether the resulting number differed between the 
early RTT and non-RTT states, and between the later RTT and non-RTT states. In other words, 
this analysis tested whether there were differences between RTT and non-RTT states with 
respect to changes between 2007-2008 and 2011-2012 in the number of policies and practices 
used. The results do not provide information on whether there was a change over time for the 
policies and practices for which we did not measure usage in 2007-2008. 

3. ELL-Focused Analyses 

We examined the extent to which states focused on ELLs in their usage of RTT-promoted 
policies and practices using the same processes that we describe above to analyze data from the 


(continued) 

school year, specifically asking if respondents found it difficult or had any problems recalling that period. All 
respondents reported that they were able to accurately answer questions about that time period. Further, interviewers 
reported that respondents who were not in their current position during the earlier time period tended to ask 
colleagues who were knowledgeable about policies or practices at the earlier time to join the interview or to provide 
responses prior to the interview. 

22 When developing the interview protocol, we included baseline questions for particular policies and practices 
based on U.S. Department of Education guidance about those in which it had interest in assessing change. 
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state interviews, except that the summary measures included only policies and practices that 
explicitly focused on ELLs. We used a systematic approach to select ELL-focused interview 
questions that aligned with the policies and practices in the RTT application criteria, in which 
two Mathematica researchers selected questions independently and a third resolved any 
differences of opinion between them (the agreement rate was 95 percent). 

To examine whether usage of these ELL-focused policies and practices differed by the size 
of the ELL population and the ELL/non-ELL achievement gap, we categorized each state 
according to whether it had an above-median or below-median ELL population and an above- or 
below-median ELL/non-ELL achievement gap, where ELL population is defined as the 
percentage of students who are ELLs. We then examined the usage of ELL-focused policies and 
practices for those groups (above-median ELL population, below-median ELL population, 
above-median gap, below-median gap). Throughout this report, we use “states with higher ELL 
populations” to refer to states with above-median ELL populations, “states with higher ELL/non- 
ELL achievement gaps” to refer to states with above-median ELL/non-ELL achievement gaps, 
“states with lower ELL populations” to refer to states with below-median ELL populations, and 
“states with lower ELL/non-ELL achievement gaps” to refer to states with below-median 
ELL/non-ELL achievement gaps. 

We classified states as having higher (above-median) or lower (below-median) ELL 
populations using CCD data on the percentage of public school students participating in 
programs for ELLs in 2007-2008. This data element was available for all 50 states and the 
District of Columbia. We classified states as having higher or lower ELL/non-ELL achievement 
gaps based on their gaps on the spring 2007 NAEP 4th grade math exam. 23 Specifically, we 
calculated the ELL/non-ELL achievement gap as average achievement for non-ELLs minus 
average achievement for ELLs. We used CCD data from 2007-2008 and NAEP data from spring 
2007 because in later years, the American Recovery and Reinvestment Act had been announced, 
and it may have motivated states to start taking steps to increase the competitiveness of their 
RTT applications, possibly affecting their ELL population or ELL/non-ELL achievement gap. 

We ran statistical tests to detennine whether there were differences in the number of ELL- 
focused policies and practices used between each of the RTT groups and the non-RTT group of 
states. Lor example, we compared early RTT states with lower ELL populations and ELL/non- 
ELL achievement gaps to non-RTT states that also had lower populations and gaps. We also ran 
statistical tests to determine whether there were differences in the number of ELL-focused 
policies and practices within each group of states. Lor example, we compared early RTT states 
with lower populations and gaps to early RTT states with higher populations and gaps. 


23 We focused on the NAEP 4th grade math exam because compared with NAEP exams for other grades and 
subjects, the 4th grade math exam had the fewest states with a missing value for this achievement gap in 2007. The 
ELL/non-ELL achievement gaps on these exams were all highly correlated (the correlations ranged from 0.7 to 0.9), 
so the choice of subject and grade was unlikely to make a large difference in the composition of the lower and 
higher groups. 
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III. UNDERSTANDING THE CONTEXT FOR USAGE OF POLICIES AND PRACTICES 

PROMOTED BY RACE TO THE TOP 

As described in Chapter I, RTT sought to promote and facilitate systemic changes at all 
levels of the U.S. educational system by supporting policies in six broad areas: (1) state capacity 
for reform; (2) rigorous standards and aligned assessments; (3) comprehensive data systems; (4) 
effective teachers and principals; (5) turnaround of low-perfonning schools; and (6) charter 
schools. The RTT grant competition, which was open to all states, the District of Columbia, and 
Puerto Rico, was designed to encourage and reward states that demonstrated past improvements 
in student achievement and presented comprehensive plans in their applications for 
implementing refonns in the areas outlined in the application. Only four states (Alaska, North 
Dakota, Texas, Vermont) did not compete for RTT in any round, and it is well documented that 
after RTT was announced, many states took steps to become eligible for the grants and to make 
their applications more competitive. 24 Therefore, when examining RTT implementation, it is 
essential to understand the education policies and practices that existed in RTT states and non- 
RTT states before the RTT program was announced , 25 This is important to ensure that we do not 
draw misleading conclusions from our study’s findings. For example, any differences found 
between RTT and non-RTT states in spring 2012 could be due to the effect of RTT, but they also 
could reflect preexisting differences between these groups of states. 

In this chapter, we compare RTT states and non-RTT states to examine the extent to which 
the groups were similar or different prior to RTT. Baseline characteristics of the three groups of 
states — (1) early RTT states, (2) later RTT states, and (3) non-RTT states — are shown in 
Table III. 1 . As discussed in Chapter II, the RTT analyses in this report are based on comparisons 
between groups (1) and (3) and between groups (2) and (3). Therefore, in the text that follows, 
we focus on contrasts between these groups of states. 


24 For instance, Illinois lifted caps on the number of charter schools it would allow, and Massachusetts made it easier for 
students in low-perfonning schools to switch to charters (National Alliance for Charter Schools, 2009); West Virginia proposed a 
merit pay system that included student achievement in compensation calculations (Council of State Governments, October 2011; 
March 2010). To be eligible to apply for RTT, states could not have laws prohibiting the linking of student achievement and 
teacher administrative records, and some states (for example, California and Maine) changed their laws to be eligible (GAO 
2011). Most states also adopted the Common Core State Standards. 

25 RTT was first announced in July 2009, with the first round of state grant applications due to ED in January 2010. 
Therefore, for most of the state characteristics examined, we focus on information from the 2007-2008 school year. One 
exception to this rule is that we examine whether states had received any State Longitudinal Data Systems (SLDS) Grants 
between 2005 and 2009, and the cumulative amount of funding received during this period. Because states would have had to 
apply for these grants prior to July 2009, this grant funding would have reflected state efforts to improve their longitudinal data 
systems before RTT was first announced. 
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Table 111.1. Baseline Characteristics of Early RTT States, Later RTT States, and Non-RTT States 




Later RTT 

Non-RTT 


Early RTT States 

States 

States 

RTT Round 1 Application Scores, Among States That Applied in Round 1 a 

Average RTT Round 1 application scores 

419* 

388* 

323 

Average RTT Round 1 application scores on 
the state reform conditions criteria 

218* 

205* 

173 

Average RTT Round 1 application scores on 
reform plan criteria 

200* 

184* 

149 

National Assessment of Educational Progress (NAEP) Scores, 2007 

Average scores: math, 4th grade 

238 

238 

240 

Average scores: math, 8th grade 

278 

281 

282 

Average scores: reading, 4th grade 

220 

220 

221 

Average scores: reading, 8th grade 

260 

262 

263 

State Longitudinal Data System (SLDS) Grant Information 

Percentage of states that received SLDS 
grants in 2005, 2007, or 2009 

91.7 

85.7 

77.4 

Average number of SLDS grants received from 
2005 to 2009 among states that did and did not 
receive these grants 13 

1.2 

1.1 

1.0 

Average total SLDS funding received from 
2005 to 2009 among states that did and did not 
receive these grants 

$5.4 million 

$6.0 million 

$4.7 million 

State Fiscal Stabilization Fund (SFSF) Grant Information 

Percentage of states that received SFSF 
grants in 2009 

100 

100 

100 

Average 2009 SFSF grant amount 

$1.16 billion 

$1.20 billion 

$0.69 billion 

State-Reported Usage of Policies and Practices, from Interview Responses (2007-2008)° 

Average number of policies and practices 
aligned with the RTT application criteria in the 
area of teacher and principal certification and 
evaluation (out of 12) 

Average number of policies and practices 

4.6* 

4.5* 

2.7 

aligned with the RTT application criteria in the 
area of school turnaround (out of 5) 

1.3 

1.6* 

0.8 

Average number of policies and practices 
aligned with the RTT application criteria in the 
area of charter schools (out of 3) 

2.0 

2.1 

1.3 

State Education Characteristics, from the Common Core of Data (2007-2008) 

Student’s race/ethnicity (average percentage) 




White, non-Hispanic 

50.9* 

60.6 

70.3 

Black, non-Hispanic 

26.6* 

16.9 

11.0 

Hispanic 

11.6 

17.3 

11.2 

Asian 

9.2 

3.4 

3.1 

Other 

1.7 

1.9 

4.4 

Average percentage of students eligible for 
free or reduced-price lunch 

40.6 

40.4 

39.7 
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Table 1 1 1.1 (continued) 



Early RTT States 

Later RTT 
States 

Non-RTT 

States 

Average percentage of schools eligible for 

Title 1 

68.8 

70.4 

63.1 

State’s region (percentage) 

Northeast 

25.0 

28.6 

12.9 

Midwest 

8.3 

14.3 

32.3 

South 

58.3* 

28.6 

22.6 

West 

8.3 

28.6 

32.3 

School level (average percentage) 

Elementary 

61.0 

61.2 

57.8 

Middle 

18.8 

18.0 

17.8 

High 

15.9* 

17.4 

19.7 

Other 

4.4 

3.5 

4.6 

Average number of full-time staff 

153,000 

153,000 

87,000 

Number of States 

11-12 

7 

23-31 


Sources: U.S. Department of Education; NAEP scores; interviews with state administrators in spring 

2012; Common Core of Data. 

Note: The characteristics reported in this table are unweighted state averages. Texas did not 

participate in the spring 2012 interviews and is excluded from this table. A range is provided 
for the sample sizes because nonresponse varied across items. 
a Alaska, North Dakota, Texas, and Vermont did not apply for RTT grants in any round and are excluded 
from this section of the table because they have no application scores. 

b States were able to receive more than one SLDS grant across the three rounds of SLDS awards in 
2005, 2007, and 2009. 

c We identified the spring 2012 state interview questions that asked about policies and practices in place 
during 2007-2008 that were aligned with the requirements laid out in the RTT application criteria. A 
higher number indicates that more policies and practices promoted by RTT were reported by the state as 
being in place in school year 2007-2008. All data provided by states were self-reported and not 
independently verified by the research team. For these reasons and potential concerns about recall 
accuracy, readers should exercise caution when interpreting data from 2007-2008. 

d More specifically, this is the average number of full-time equivalent staff, which means that staff who 
work part time are counted as a proportion of a staff member, rather than as a whole staff member. For 
example, two half-time employees are counted as one full-time equivalent staff member. The average 
numbers of full-time equivalent staff were rounded to the nearest thousand. 

*Significantly different from non-RTT states at the .05 level, two-tailed test. 

The groups of states differed by a statistically significant margin at baseline on the 
following characteristics; 

• RTT Round 1 application scores. As one might expect, the average total scores 
received from grant application reviewers for the Round 1 applications of both early 
and later RTT states (419 and 388) were statistically significantly higher than the 
average scores for non-RTT states that applied for but did not win RTT grants 
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(323). 26 A maximum score of 500 points was possible. 27 More than half of the points 
that a state’s application could receive (260) were based on demonstrated 
accomplishments prior to applying — for example, increasing student achievement, 
enlisting statewide support for proposed plans, or creating legal conditions conducive 
to policy changes. Compared to non-RTT states that applied for RTT, both early and 
later RTT states had statistically significantly higher scores on these accomplishments 
prior to applying (referred to as “state reform conditions” in the application criteria). 

The higher scores suggest potentially important preexisting differences between these 
groups of states. The balance of points that a state’s application could receive was 
awarded based on the state’s proposed policy plans. Both early and later RTT states 
had statistically significantly higher scores than non-RTT states that applied for RTT 
on these plans (referred to as “state refonn plans” in the application criteria). 

• Policies and practices for teacher and principal certification and evaluation and 
for school turnaround. Both early and later RTT states reported having statistically 
significantly more policies and practices aligned with RTT in the 2007-2008 baseline 
year in the area of teacher and principal certification and evaluation (4.6 for early 
RTT states, 4.5 for later RTT states, 2.7 for non-RTT states). Later RTT states 
reported having statistically significantly more policies and practices aligned with 
RTT in the 2007-2008 baseline year in the school turnaround area (1.6 for later RTT 
states, 0.8 for non-RTT states). As noted in Chapter II, 2007-2008 findings should be 
interpreted with caution because data from 2007-2008 were collected retrospectively 
in spring 2012. 

• State education characteristics. Compared to non-RTT states at baseline, according 
to the Common Core of Data for 2007-2008, early RTT states had a higher 
percentage of students who were black (26.6 vs. 1 1.0 percent), a lower percentage of 
students who were white (50.9 vs. 70.3 percent), were more likely to be located in the 
south (58.3 vs. 22.6 percent), and had a smaller percentage of high schools (15.9 vs. 
19.7 percent). Differences in all these characteristics were statistically significant. 

The groups of states did not differ to a statistically significant degree on the following 
baseline characteristics: 

• Average 2007 NAEP scores for students in 4th grade and 8th grade in both 
reading and mathematics. 

• Percentage of states that received at least one SLDS grant between 2005 and 
2009, average number of SLDS grants received over that time period, and 
average amount of total SLDS grant funding received over that time period. 28 


26 We focus on Round 1 scores in order to use scores from the greatest number of applicants. We would be 
missing 1 1 states if we used Round 2 scores (i.e., the 2 Round 1 states and 9 states that applied in Round 1 only), 
whereas using Round 1 scores eliminates 6 states that only applied in Round 2. 

27 See http://www2.ed.gov/programs/racetothetop/scoringrubric.pdf. 

28 States were able to receive more than one SLDS grant across the three rounds of SLDS awards in 2005, 
2007, and 2009. 
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• Percentage of states that received an SFSF grant and average amount of SFSF 
grant funding received. 29 

• Average number of policies and practices aligned with the RTT application 
criteria in the 2007-2008 baseline year in the area of charter schools. Data from 
2007-2008 were collected retrospectively in spring 2012. All data provided by states 
were self-reported and not independently verified by the research team. For these 
reasons and potential concerns about recall accuracy, readers should exercise caution 
when interpreting data from 2007-2008. 

• State education characteristics. Later RTT states and non-RTT states did not differ 
by a statistically significant margin on any of the variables examined from the 
Common Core of Data for 2007-2008. Early RTT states and non-RTT states did not 
differ by a statistically significant margin in terms of the percentage of students who 
were Hispanic, Asian, or other race; the percentage of students eligible for free or 
reduced-price lunch; the percentage of schools eligible for Title I; the percentage of 
states from the Northeast, Midwest, and West; the percentage of elementary, middle, 
and other schools; or the number of full-time staff. 

These results suggest that some important baseline differences existed between the groups of 
states before RTT efforts got underway. When interpreting this study’s RTT findings, it will be 
important to keep these differences in mind. Any statistically significant differences in policies 
and practices observed in spring 2012 may not necessarily be attributable to the RTT program 
and instead might reflect preexisting differences between the groups. 


29 It is important to understand whether RTT and non-RTT states differed in their receipt of SFSF grants and 
the amount of SFSF grant funding received because the SFSF grants were large grants (the average 2009 SFSF grant 
amount for early RTT states was $1.16 billion) awarded to governors in exchange for a commitment to advance 
many of the same types of education reforms that were promoted by RTT, such as college- and career-ready 
standards and high-quality, valid, and reliable assessments for all students; development and use of pre-K through 
post-secondary and career data systems; increasing teacher effectiveness and ensuring an equitable distribution of 
qualified teachers; and turning around the lowest-performing schools. 
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IV. STATES’ USAGE OF POLICIES AND PRACTICES 
PROMOTED BY RACE TO THE TOP 

To understand the ultimate effectiveness of a grant program like RTT in improving student 
achievement, it is important first to understand the extent to which the policies and practices it 
promotes are being used by RTT states. If none (or very few) of the RTT states used these 
policies and practices, it is unlikely that any changes in outcomes — positive or negative — could 
be attributed to the program. In this chapter, we assess the extent to which RTT states and non- 
RTT states reported using policies and practices promoted by RTT. 

As noted previously, RTT promoted specific policies in six topic areas. In this chapter, we 
summarize the extent to which state administrators reported in spring 2012 using policies and 
practices aligned with each of these topic areas and their subtopics. Appendix A presents more 
detailed findings for the subtopics. 

A. State Capacity 

A premise of RTT is that state education agencies can play a direct role in advancing 
statewide education policies. Thus, RTT aims to improve state capacity to support education 
policy. Section A of the RTT application, state success factors, focused on three subtopics: (1) 
articulating the state’s education reform agenda and LEAs’ participation in it, (2) building strong 
statewide capacity to implement, scale up, and sustain the proposed plans; and (3) demonstrating 
significant progress in raising achievement and closing gaps. We identified 10 policies and 
practices from the spring 2012 state interview aligned with RTT objectives on state capacity 
(Table IV. 1). 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the state capacity criteria in the RTT application than 
did non-RTT states. Early RTT states reported using an average of 7.5 out of 10 policies and 
practices in this area, compared to 6.2 for non-RTT states (Figure IV. 1). As baseline data about 
policies and practices aligned with RTT objectives on state capacity were not collected, we could 
not detennine whether this difference existed before the awarding of RTT grants. 


30 Because no state interview items aligned to subtopic 3, it is excluded from the analysis. 
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Table IV.1. Policies and Practices Aligned with RTT Objectives on State Capacity, by Subtopic 

Articulating the state’s education reform agenda and LEAs’ participation in it 

Having a comprehensive education reform plan in place 

Prioritizing the adoption and implementation of college and career ready standards or assessments 

Prioritizing the development of comprehensive, student-level, longitudinal data systems or using data to improve 
instruction 

Prioritizing the recruiting, developing, rewarding, and retaining of effective teachers and principals, especially 
where they are needed most (including developing and implementing evaluation systems) 

Prioritizing the turnaround of its lowest-achieving schools 

Building strong statewide capacity to implement, scale up, and sustain the proposed plans 

Providing leadership or teams to support the implementation of reform plans 

Working with districts that have the highest achievement levels and seeking to replicate their practices statewide to 
implement the education reform plan 

Providing effective and efficient operations and processes for grant oversight and performance measure tracking 
and reporting 

Involving teachers, teachers unions or associations, or school administrators in defining its education reform 
initiatives or priorities 

Involving other stakeholders in defining its education reform initiatives or priorities 

Source: Section A of the RTT application (http://www2.ed.gov/programs/racetothetop/phase2-application.doc); 

interviews with state administrators in spring 2012. 

Note: The spring 2012 state interview asked only whether these policies and practices were in place during 

the current school year (2011-2012). See Appendix C for a list of the specific interview questions that 
were aligned with the RTT policies and practices in this table. See the RTT application for a detailed 
description of each subtopic. 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the RTT application criteria than did non-RTT states 
for both state capacity subtopics: 

• Articulating the state’s education reform agenda and LEAs’ participation in it. 

Early RTT states reported using an average of 4.2 out of 5 policies and practices in 
this area, compared to 3.6 for non-RTT states (Appendix A Figure A.l). 

• Building strong statewide capacity to implement, scale up, and sustain the 
proposed plans. Early RTT states reported using an average of 3.2 out of 5 policies 
and practices in this area, compared to 2.6 for non-RTT states (Appendix A Figure 
A. 2). 
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Figure IV.1. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
State Capacity, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV.1. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 
10 examined) that were aligned with the RTT application criteria. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. For 
example, three non-RTT states reported using 5 of the 10 state capacity policies and practices aligned 
with the RTT application criteria. For 3 of the policies and practices, a “yes” response received one 
point. In the other 7 cases, it was possible for a state to receive a fraction of one point. See Chapter II 
for details on the way in which the number of policies and practices was determined for each state. The 
dashed line denotes the average number of policies and practices for each group of states. 

"Significantly different from non-RTT states at the 0.05 level, two-tailed test. 

The individual policy or practice with the highest level of usage in the state capacity 
area was having a comprehensive education reform plan in place. Across all states, the 
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3 1 

average value for this practice was 0.92. Average values for each group of states (early RTT, 
later RTT, non-RTT) are shown in Figure IV.2. 


Figure IV.2. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on State Capacity, Articulating State’s Reform Agenda Subtopic, Spring 2012 


Articulating State's Reform Agenda 



ro 0.0 

a> 


Extent to Extent to Extent to 

Which State Which State Which State 

Prioritizes Prioritizes Prioritizes 

CCR Standards Development of Teacher and 

or Assessments SLDS or Principal 

Data Use Evaluation and 

Effectiveness 

Policy or Practice Aligned with RTT Objectives 


| Early RTT States (N=12)^Later RTT States (N=7)|| Non-RTT States (N=31) 

Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). 

CCR = college and career ready; SLDS = state longitudinal data system. 


31 As described in Chapter II, for each policy or practice in the RTT application criteria for which we identified 
one or more relevant interview questions, we constructed a variable ranging from zero to one using those questions. 
To calculate the average value of a policy or practice, we averaged that variable across all states. For example, if 
half the states used a particular practice (meaning they had a value of 1) and the other half of states did not use that 
practice (meaning they had a value ofO), the average value for that practice would be 0.5. 
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The individual policies and practices with the lowest levels of usage in the state 
capacity area were (1) involving teachers, teachers unions or associations, or school 
administrators in defining education reform initiatives or priorities; and (2) involving other 
stakeholders in defining education reform initiatives or priorities. Across all states, the 
average values for these policies and practices were 0.29 and 0.25. Average values for each 
group of states (early RTT, later RTT, non-RTT) are shown in Figure IV. 3. 

Figure IV.3. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on State Capacity, Building Strong Statewide Capacity Subtopic, Spring 2012 


Building Strong Statewide Capacity 



Extent to 
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Implementation 
of Reform Plan 
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Extent to Extent to 

Which State Which State 
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Unions, or Define Priorities 

Administrators 
to Define 
Priorities 

Policy or Practice Aligned with RTT Objectives 



| Early RTT States (N=12)^Later RTT States (N=7)^ Non-RTT States (N=27-31) 


Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). A range is provided for the sample sizes because nonresponse varied across items. 


B. Standards and Assessments 

One goal of RTT is for states to adopt standards and assessments that prepare students to 
succeed in college and the workplace, and to compete in the global economy. Section B of the 
RTT application, standards and assessments, focused on three subtopics: (1) developing and 
adopting common standards; (2) developing and implementing common, high-quality 
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assessments; and (3) supporting the transition to enhanced standards and high-quality 
assessments. We identified 10 policies and practices from the spring 2012 state interview aligned 
with RTT objectives on standards and assessments (Table IV. 2). 

Table IV.2. Policies and Practices Aligned with RTT Objectives on Standards and Assessments, by Subtopic 

Developing and Adopting Common Standards 

Adopting the Common Core State Standards (CCSS) in both English/language arts and math 

Developing and Implementing Common, High Quality Assessments 

Participating in a consortium of states to develop assessments aligned to CCSS 

Has specified the school year by which state plans to fully implement summative assessments being developed by 
the consortium 

Supporting the Transition to Enhanced Standards and High-Quality Assessments 

Supporting the implementation of CCSS by providing funds for additional staff or making new technology 
investments 

Supporting the implementation of the assessments associated with the CCSS by making new technology 
investments 

Requiring or supporting the use of new instructional materials for implementing the CCSS 
Developing, supporting, or requiring new interim assessments associated with the CCSS 
Changing high school exit requirements or college entrance requirements 

Changing policies about or providing funds for professional development, training, and technical assistance for 
teachers or school principals to support implementation of CCSS 

Supporting districts or schools in implementing the CCSS with English language learners 

Source: Section B of the RTT application (http://www2.ed.gov/programs/racetothetop/phase2-application.doc); 

interviews with state administrators in spring 2012. 

Note: The spring 2012 state interview asked only whether these policies and practices were in place during the 

current school year (2011-2012). See Appendix C for a list of the specific interview questions that were 
aligned with the RTT policies and practices in this table. See the RTT application for a detailed description 
of each subtopic. 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the standards and assessments criteria in the RTT 
application than did non-RTT states. Early RTT states reported using an average of 5.9 out of 
10 policies and practices in this area, compared to 4.0 for non-RTT states (Figure IV. 4). As 
baseline data about policies and practices aligned with RTT objectives on standards and 
assessment were not collected, we could not determine whether this difference existed before the 
awarding of RTT grants. 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the RTT application criteria than did non-RTT states 
for one of the three standards and assessment subtopics. Early RTT states reported using an 
average of 3.1 out of 7 policies and practices in the subtopic focused on supporting the transition 
to enhanced standards and high-quality assessments, compared to 1.6 for non-RTT states 
(Appendix A Figure A. 5). 
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Figure IV.4. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Standards and Assessments, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 2. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 
10 examined) that were aligned with the RTT application criteria. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. For 
example, two early RTT states reported using 5 of the 10 standards and assessment policies and 
practices aligned with the RTT application criteria. For 5 of the policies and practices, a “yes” response 
received one point. In the other 5 cases, it was possible for a state to receive a fraction of one point. 
See Chapter II for details on the way in which the number of policies and practices was determined for 
each state. The dashed line denotes the average number of policies and practices for each group of 
states. 

"Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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The individual policy or practice with the highest level of usage in the standards and 
assessments area was specifying the school year by which the state plans to fully implement 
summative assessments being developed by the consortium. Across all states, the average 
value for this practice was 0.94. Average values for each group of states (early RTT, later RTT, 
non-RTT) are shown in Figure IV. 5. 

Figure IV.5. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on Standards and Assessments, Developing and Adopting Common Standards 
Subtopic and Developing and Implementing Common, High-Quality Assessments Subtopic, 
Spring 2012 


Developing and Adopting 
Common Standards 


Developing and Implementing 
Common, High-Quality Assessments 
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Policy or Practice Aligned with RTT Objectives 


Fully Implement 
Assessments Developed 
by Consortium 


| Early RTT States (N=11-12)^Later RTT States (N=7)J Non-RTT States (N=28-31) 


Source: Interviews with state administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each policy or 

practice in the RTT application criteria for which we identified one or more interview questions aligned 
with the policy or practice, we constructed a variable ranging from zero to one, with a value of one 
indicating that the state responded “yes” to all the interview questions selected for that policy or 
practice. The height of each bar represents the mean value of the policy or practice (on a scale of zero 
to one) for each group of states. A range is provided for the sample sizes because nonresponse varied 
across items. 

CCSS = Common Core State Standards. 


The individual policies and practices with the lowest levels of usage in the standards 
and assessments area were (1) developing, supporting, or requiring new interim 
assessments associated with the CCSS; and (2) changing high school exit requirements or 
college entrance requirements. Across all states, the average value for both of these policies 
and practices was 0.09. Average values for each group of states (early RTT, later RTT, non- 
RTT) are shown in Figure IV. 6. 
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Figure IV.6. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on Standards and Assessments, Supporting the Transition to Enhanced Standards and 
High-Quality Assessments Subtopic, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; ELLs = English language learners; PD = professional development. 


C. Data Systems 

One goal of RTT is for states to develop data systems that measure student achievement 
growth and success, and to use those systems to inform teachers’ and principals’ efforts to 
improve instruction. Section C of the RTT application, data systems to improve instruction, 
focused on three subtopics: (1) fully implementing an SLDS, (2) accessing state data and using it 
to infonn key stakeholders, and (3) using data to improve instruction. We identified eight 
policies and practices from the spring 2012 state interview aligned with RTT objectives on data 
systems (Table IV. 3). 
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Table IV.3. Policies and Practices Aligned with RTT Objectives on Data Systems, by Subtopic 

Fully Implementing an SLDS 

Having an SLDS 

SLDS contains program participation information 3 
SLDS is linked to an early childhood data system 
SLDS is linked to a higher education data system 

Accessing State Data and Using It to Inform Key Stakeholders 

SLDS can be accessed by stakeholders 

SLDS is used to inform and engage stakeholders and support decision makers in continuous improvement efforts 

Using Data to Improve Instruction 
Requiring districts to implement district data systems 15 

Providing funding, materials, training, technical assistance, or other supports to districts to encourage the use of 
data to improve instruction 

Source: Section C of the RTT application (http://www2.ed.gov/programs/racetothetop/phase2-application.doc); 

interviews with state administrators in spring 2012. 

Note: The spring 2012 state interview asked only whether these policies and practices were in place during 

the current school year (2011-2012). See Appendix C for a list of the specific interview questions that 
were aligned with the RTT policies and practices in this table. See the RTT application for a detailed 
description of each subtopic. 

3 The interview questions comprising this policy or practice focused primarily on ELLs. See Appendix C for the 
interview questions that comprised each policy or practice. 

b District data systems, which are also called local instructional improvement systems, are defined by the U.S. 
Department of Education in the RTT application materials as technologically based tools or strategies that provide 
educators with data to manage continuous instructional improvement efforts. 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the data systems criteria in the RTT application than 
did non-RTT states. Early RTT states reported using an average of 6.3 out of 8 policies and 
practices in this area, compared to 5.1 for non-RTT states (Figure IV. 7). As baseline data about 
policies and practices aligned with RTT objectives on data systems were not collected, we could 
not determine whether this difference existed before the awarding of RTT grants. 

The individual policy or practice with the highest level of usage in the data systems 
area was having an SLDS. Across all states, the average value for this practice was 0.94. 
Average values for each group of states (early RTT, later RTT, non-RTT) are shown in 
Figure IV. 8. 

The individual policy or practice with the lowest level of usage in the data systems area 
was requiring districts to implement district data systems. Across all states, the average value 
for this policy was 0.30. Average values for each group of states (early RTT, later RTT, non- 
RTT) are shown in Figure IV. 8. 
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Figure IV.7. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on Data 
Systems, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 3. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 
eight examined) that were aligned with the RTT application criteria. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. For 
example, one non-RTT state reported using three of the eight data systems policies and practices 
aligned with the RTT application criteria. For five of the policies and practices, a “yes” response 
received one point. In the other three cases, it was possible for a state to receive a fraction of one point. 
See Chapter II for details on the way in which the number of policies and practices was determined for 
each state. The dashed line denotes the average number of policies and practices for each group of 
states. 

"Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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Figure IV.8. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on Data Systems, Spring 2012 


Fully Implementing a SLDS 


Using Data to 
Improve Instruction 



t r 

Have SLDS Extent to SLDS Linked SLDS Linked Extent to Which Extent to Extent to Which Provide 
Which SLDS to EC System to HE System State Allows Which State State Requires Supports 

Includes SLDS to Uses SLDS Districts to to Districts 

Program be Accessed to Support Implement Data for Using 

Participation by Stakeholders Decision System Data 

Data Makers 

Policy or Practice Aligned with RTT Objectives 


| Early RTT States (N=1 1-1 2)^ Later RTT States (N=7)^Non-RTT States (N=31) 


Source: Interviews with state administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each policy or 

practice in the RTT application criteria for which we identified one or more interview questions aligned 
with the policy or practice, we constructed a variable ranging from zero to one, with a value of one 
indicating that the state responded “yes” to all the interview questions selected for that policy or 
practice. The height of each bar represents the mean value of the policy or practice (on a scale of zero 
to one) for each group of states. For some of the policies and practices shown in this figure, multiple 
interview questions aligned with that policy or practice. In the figure, we indicate this using the words 
“extent to which” at the beginning of the policy or practice, to emphasize that the level of usage of that 
policy or practice is measured using multiple interview questions (as opposed to a single, binary 
measure of whether that policy or practice was used). A range is provided for the sample sizes because 
nonresponse varied across items. 

EC = early childhood; HE = higher education; SLDS = statewide longitudinal data system. 


D. Teacher and Principal Certification and Evaluation 

One goal of RTT is to recruit, develop, reward, and retain effective teachers and principals, 
especially in the high-poverty schools and districts where they are needed most. Section D of the 
RTT application focused on teacher and principal certification and evaluation, with specific 
emphasis on five subtopics: ( 1 ) providing high-quality pathways to certification for aspiring 
teachers and principals, (2) improving teacher and principal effectiveness based on performance, 
(3) ensuring equitable distribution of effective teachers and principals, (4) improving the 
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effectiveness of teacher and principal preparation programs, and (5) providing effective support 
to teachers and principals. 32 We identified 39 policies and practices from the spring 2012 state 
interview aligned with RTT objectives on teacher and principal certification and evaluation 
(Table IV.4). 

Table IV.4. Policies and Practices Aligned with RTT Objectives on Teacher and Principal Certification and 
Evaluation, by Subtopic 

Providing High-Quality Pathways to Certification for Aspiring Teachers and Principals 

Authorizing or expanding institutions qualified to operate alternative-route programs for teachers or principals 3 
Adopting policies to increase the selectivity of alternative-route programs for teachers or principals 

Adopting policies to increase the amount of mentoring for participants in certification programs or the time such 
participants spend in school-based learning experiences for teachers or principals 

Adopting policies allowing alternative-route programs to award the same type of certification as traditional preparation 
programs for teachers or principals 

Groups are currently operating alternative-route programs for teachers or principals 3 
Having a process to identify areas of shortages for teachers or principals 
Taking steps to address areas of shortage for teachers or principals 

Improving Teacher and Principal Effectiveness Based on Performance 

Requiring districts to use student growth in evaluations and specifying the extent to which student achievement growth 
must factor into evaluations of teachers or principals 3 

Requiring multiple performance measures for evaluations of teachers or principals 3 

Specifying a required minimum number of rating categories to be used when evaluating teachers or principals 3 
Conducting annual evaluations of teachers 3,b 

Requiring evaluations to inform decisions about professional development and support for teachers or principals 
Requiring evaluations to inform decisions about compensation for teachers or principals 
Requiring evaluations to inform decisions about career advancement for teachers or principals 
Requiring evaluations to inform decisions about dismissal of teachers or principals 

Ensuring Equitable Distribution of Effective Teachers and Principals 

Requiring districts or schools to use strategies to promote a more equitable distribution of effective teachers or principals 

Conducting analyses of effectiveness based on student achievement growth to determine whether there has been a shift 
in the distribution of effective teachers or principals 

Improving the Effectiveness of Teacher and Principal Preparation Programs 

Using effectiveness ratings (based in part on student achievement growth) to assess the effectiveness of certification 
programs for teachers or principals 3 

Publicly reporting results from its evaluations of certification program effectiveness for teachers or principals 

Using results from its evaluations of certification programs to provide additional funds for, expand, or promote certification 
programs that were shown to be effective for teachers or principals 

Source: Section D of the RTT application (http://www2.ed.gov/programs/racetothetop/phase2-application.doc); interviews 

with state administrators in spring 2012. 

Note: The phrase “teachers or principals” indicates that we analyzed two separate practices: one focused on teachers and 

one focused on principals. Unless otherwise noted, the spring 2012 state interview asked only whether these policies 
and practices were in place during the current school year (2011-2012). See Appendix C for a list of the specific 
interview questions that were aligned with the RTT policies and practices in this table. See the RTT application for a 
detailed description of each subtopic. Most of the rows in this table describe two separate practices (one for 
teachers, one for principals). 


32 


Because no state interview items aligned to subtopic 5, it is excluded from the analysis. 
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Table IV. 4 (continued) 

a The spring 2012 state interview asked not only whether this policy or practice was in place during the current school year 
(2011-2012) but also whether it was in place during the 2007-2008 school year. 
b The state interview did not ask about this policy or practice for principals. 

In spring 2012, both early and later RTT states reported using statistically significantly 
more of the policies and practices aligned with the teacher and principal certification and 
evaluation criteria in the RTT application than did non-RTT states. Early and later RTT 
states reported using averages of 13.6 and 11.2 of the 39 policies and practices in this area, 
compared to 7.4 for non-RTT states (Figure IV. 9). However, for both early and later RTT states, 
baseline data on 12 of these 39 policies and practices show that a statistically significant 
difference between the RTT and non-RTT states existed prior to the awarding of RTT grants. In 
particular, in 2007-2008, RTT states reported, on average, having 4.6 and 4.5 of these 12 
policies and practices in place, compared to 2.7 for non-RTT states. (Baseline data were not 
collected for the other 27 policies and practices examined, so we could not determine if there 
were baseline differences for the topic area as a whole.) 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the RTT application criteria than did non-RTT states 
for three of the four teacher and principal certification and evaluation subtopics: 

• Providing high-quality pathways to certification for aspiring teachers and 
principals. Early RTT states reported using an average of 6.4 out of 14 policies and 
practices in this area, compared to 3.7 for non-RTT states (Appendix A Figure A. 9). 

• Improving teacher and principal effectiveness based on performance. Early RTT 
states reported using an average of 6.1 out of 15 policies and practices in this area, 
compared to 3.4 for non-RTT states (Appendix A Figure A. 10). 

• Ensuring equitable distribution of effective teachers and principals. Early RTT 
states reported using an average of 0.6 out of 4 policies and practices in this area, 
compared to 0.2 for non-RTT states (Appendix A Figure A. 1 1). 
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Figure IV.9. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Teacher and Principal Certification and Evaluation, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV.4. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 
39 examined) that were aligned with the RTT application criteria. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. For 
example, four non-RTT states reported using 10 of the 39 teacher and principal certification and 
evaluation policies and practices aligned with the RTT application criteria. For 25 of the policies and 
practices, a “yes” response received one point. In the other 14 cases, it was possible for a state to 
receive a fraction of one point. See Chapter II for details on the way in which the number of policies and 
practices was determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

"Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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The increases over time in the usage of teacher evaluation policies and practices 
aligned with the RTT application criteria were not statistically significantly different 
between RTT and non-RTT states. We examined the 12 teacher and principal certification and 
evaluation policies and practices aligned with the RTT application criteria for which we had state 
reports for both the 2007-2008 and 2011-2012 school years. Statistically significant differences 
between eventual RTT states and non-RTT states were already evident in 2007-2008, and the 
usage of these policies and practices increased within each group between 2007-2008 and 20 11— 
2012 (Figure IV. 10). However, the increases over time in usage of these 12 policies and practices 
for early and later RTT states did not differ significantly from the increase over time for non- 
RTT states. As noted in Chapter II, these findings about changes between 2007-2008 and 20 11— 
2012 should be interpreted with caution because data from 2007-2008 were collected 
retrospectively in spring 2012. 

Figure IV.10. Change in Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Teacher and Principal Certification and Evaluation 



— Early RTT States (N=12) — Non-RTT States (N=31) 

— Later RTT States (N=7) 

Source: Interviews with state administrators in spring 2012. 

Note: This figure shows change over time for each group of states in the usage of policies and practices 

aligned with the teacher and principal certification and evaluation section of the RTT application criteria. 
The arrow for each group of states starts at the average number of reported policies and practices 
aligned with the RTT application criteria in 2007-2008 and ends at the average number of reported 
policies and practices aligned with the RTT application criteria in spring 2012. For example, on average, 
non-RTT states reported that they had used 2.7 of the 12 policies and practices aligned with the overall 
teacher and principal evaluation and certification topic area in 2007-2008. This group of states, on 
average, reported using 4.4 of these 12 policies and practices in spring 2012. The average usage levels 
for spring 2012 differ from those reported earlier in the chapter because this figure focuses on just the 
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Figure IV. 10 (continued) 

subset of policies and practices (12 out of 39) for which we measured usage in both 2007-2008 and 
2011-2012. There were no statistically significant differences between RTT and non-RTT states with 
respect to changes between 2007-2008 and 2011-2012 in the number of policies and practices used, 
at the 0.05 level using a two-tailed test. Data from 2007-2008 were collected retrospectively in spring 
2012. All data provided by states were self-reported and not independently verified by the research 
team. For these reasons and potential concerns about recall accuracy, readers should exercise caution 
when interpreting data from 2007-2008. 

The individual policy or practice with the highest level of usage in the teacher and 
principal certification and evaluation area was having a process to identify teacher 
shortage areas. Across all states, the average value for this practice was 0.94. Average values 
for each group of states (early RTT, later RTT, non-RTT) are shown in Figure IV. 1 1. 


Figure IV.11. States’ Reported Usage of the First Seven Individual Policies and Practices Aligned 
with RTT Objectives on Teacher and Principal Certification and Evaluation, Providing High-Quality 
Pathways to Certification for Aspiring Teachers and Principals Subtopic, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). A range is provided for the sample sizes because nonresponse varied across items. 
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The individual policies and practices with the lowest levels of usage in the teacher and 
principal certification and evaluation area were (1) requiring evaluations to inform 
decisions about career advancement for teachers; (2) requiring evaluations to inform 
decisions about career advancement for principals; (3) conducting analyses of effectiveness 
based on student achievement growth to determine whether there has been a shift in the 
distribution of effective principals; (4) using results from its evaluations of certification 
programs to provide additional funds for, expand, or promote certification programs that 
were shown to be effective for teachers; (5) publicly reporting results from its evaluations 
of certification program effectiveness for principals; and (6) using results from its 
evaluations of certification programs to provide additional funds for, expand, or promote 
certification programs that were shown to be effective for principals. Across all states, the 
average value for each of these policies and practices was 0.02. Average values for each group of 
states (early RTT, later RTT, non-RTT) are shown in Figures IV. 13, IV. 14, IV. 15, and IV. 16. 
(Figure IV. 12 shows similar information for other policies and practices.) 


Figure IV.12. States’ Reported Usage of the Last Seven Individual Policies and Practices Aligned 
with RTT Objectives on Teacher and Principal Certification and Evaluation, Providing High-Quality 
Pathways to Certification for Aspiring Teachers and Principals Subtopic, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we identified one or 

more interview questions aligned with the policy or practice, we constructed a variable ranging from zero to one, with 
a value of one indicating that the state responded “yes” to all the interview questions selected for that policy or 
practice. The height of each bar represents the mean value of the policy or practice (on a scale of zero to one) for 
each group of states. For some of the policies and practices shown in this figure, multiple interview questions aligned 
with that policy or practice. In the figure, we indicate this using the words “extent to which” at the beginning of the 
policy or practice, to emphasize that the level of usage of that policy or practice is measured using multiple interview 
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Figure IV. 12 (continued) 

questions (as opposed to a single, binary measure of whether that policy or practice was used). A range is provided 
for the sample sizes because nonresponse varied across items. 


Figure IV.13. States’ Reported Usage of the First Eight Individual Policies and Practices Aligned 
with RTT Objectives on Teacher and Principal Certification and Evaluation, Improving Teacher and 
Principal Effectiveness Based on Performance Subtopic, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). A range is provided for the sample sizes because nonresponse varied across items. 

PD = professional development. 


43 


IV. States ’ Usage of Policies and Practices Promoted by RTT Usage of Policies and Practices Promoted by RTT 


Figure IV.14. States’ Reported Usage of the Last Seven Individual Policies and Practices Aligned 
with RTT Objectives on Teacher and Principal Certification and Evaluation, Improving Teacher and 
Principal Effectiveness Based on Performance Subtopic, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). 

PD = professional development. 
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Figure IV.15. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on Teacher and Principal Certification and Evaluation, Ensuring Equitable Distribution 
of Effective Teachers and Principals Subtopic, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). A range is provided for the sample sizes because nonresponse varied across items. 
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Figure IV.16. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on Teacher and Principal Certification and Evaluation, Improving the Effectiveness of 
Teacher and Principal Preparation Programs Subtopic, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Note: As described in Chapter II, for each policy or practice in the RTT application criteria for which we 

identified one or more interview questions aligned with the policy or practice, we constructed a variable 
ranging from zero to one, with a value of one indicating that the state responded “yes” to all the 
interview questions selected for that policy or practice. The height of each bar represents the mean 
value of the policy or practice (on a scale of zero to one) for each group of states. For some of the 
policies and practices shown in this figure, multiple interview questions aligned with that policy or 
practice. In the figure, we indicate this using the words “extent to which” at the beginning of the policy or 
practice, to emphasize that the level of usage of that policy or practice is measured using multiple 
interview questions (as opposed to a single, binary measure of whether that policy or practice was 
used). 


E. School Turnaround 

Turning around the nation’s persistently lowest-achieving schools is another goal of RTT. 
Section E of the RTT application, turning around the lowest-achieving schools, focused on two 
subtopics: (1) authority to intervene in the lowest-achieving schools and LEAs, and (2) turning 
around the lowest-achieving schools. We identified 10 school turnaround policies and practices 
from the spring 2012 state interview aligned with RTT objectives on school turnaround (Table 
IV. 5). 
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Table IV.5. Policies and Practices Aligned with RTT Objectives on School Turnaround, by Subtopic 

Authority to Intervene in the Lowest-Achieving Schools and LEAs 

Having the authority to take over failing schools 3 

Turning Around the Lowest-Achieving Schools 

Providing training to the lowest-achieving schools or LEAs on analyzing student assessment data to improve 
instruction 

Helping the lowest-achieving schools or LEAs align curricula to state standards 

Providing training to the lowest-achieving schools or LEAs on identifying and implementing effective curricula, 
instructional strategies, or school intervention models, or developing and implementing a school improvement 
plan 

Providing training to the lowest-achieving schools or LEAs on identifying and implementing strategies to address 
the needs of ELLs 

Providing technical assistance to the lowest-achieving schools or LEAs on improving the quality of professional 
development 

Providing operational flexibility and support to lowest-achieving schools or LEAs with regard to staffing and 
budgeting 3 

Implementing or providing technical assistance on strategies to recruit and retain effective teachers (such as 
financial incentives) 

Having teacher tenure rules that affect placement in or removal from the lowest-achieving schools 3 
Having state-level administrative structures intended to support school turnaround efforts 3 

Source: Section E of the RTT application (http://www2.ed.gov/programs/racetothetop/phase2-application.doc); 

interviews with state administrators in spring 2012. 

Note: Unless otherwise noted, the spring 2012 state interview asked only whether these policies and practices 

were in place during the current school year (2011-2012). See Appendix C for a list of the specific 
interview questions that were aligned with the RTT policies and practices in this table. See the RTT 
application for a detailed description of each subtopic. 

3 The spring 2012 state interview asked not only whether this policy or practice was in place during the current school 
year (201 1-2012) but also whether it was in place during the 2007-2008 school year. 

ELL = English language learner; LEA = local education agency. 

In spring 2012, there were no statistically significant differences between RTT and 
non-RTT states in the school turnaround area. Early RTT states, later RTT states, and non- 
RTT states reported using averages of 6.5, 6.4, and 5.8 of the 10 policies and practices aligned 
with the school turnaround section of the RTT application (Figure IV. 17). 

The increases over time in the usage of school turnaround policies and practices 
aligned with the RTT application criteria were not statistically significantly different 
between RTT and non-RTT states. We examined the five school turnaround policies and 
practices aligned with the RTT application criteria for which we had state reports for both the 
2007-2008 and 2011-2012 school years (Figure IV. 18). The increases over time in these five 
policies and practices did not differ significantly by RTT status. As noted in Chapter II, these 
findings about changes between 2007-2008 and 2011-2012 should be interpreted with caution 
because data from 2007-2008 were collected retrospectively in spring 2012. 
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Usage of Policies and Practices Promoted by RTT 


Figure IV.17. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
School Turnaround, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 5. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of 

10 examined) that were aligned with the RTT application criteria. The number inside each dot is the 
number of states represented by the dot; dots that represent only one state have no number inside. For 
example, three non-RTT states reported using 5 of the 10 school turnaround policies and practices 
aligned with the RTT application criteria. For 6 of the policies and practices, a “yes” response received 
one point. In the other 4 cases, it was possible for a state to receive a fraction of one point. See Chapter 

11 for details on the way in which the number of policies and practices was determined for each state. 
The dashed line denotes the average number of policies and practices for each group of states. There 
were no statistically significant differences between RTT and non-RTT states at the 0.05 level using a 
two-tailed test. 
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Figure IV.18. Change in Reported Usage of Policies and Practices Aligned with RTT Objectives on 
School Turnaround 


Overall School 



Source: 

Note: 


— Early RTT States (N=12) — Non-RTT States (N=31) 

— Later RTT States (N=7) 


Interviews with state administrators in spring 2012. 

This figure shows change over time for each group of states in the usage of policies and practices 
aligned with the school turnaround section of the RTT application. The arrow for each group of states 
starts at the average number of reported policies and practices aligned with the RTT application criteria 
in 2007-2008 and ends at the average number of reported policies and practices aligned with the RTT 
application criteria in spring 2012. For example, on average, non-RTT states reported that they had 
used 0.8 of the 5 policies and practices aligned with the overall school turnaround topic area in 2007- 
2008. This group of states, on average, reported using 1.3 of these 5 policies and practices in spring 
2012. The average usage levels for spring 2012 differ from those reported earlier in the chapter 
because this figure focuses on just the subset of policies and practices (5 out of 10) for which we 
measured usage in both 2007-2008 and 2011-2012. There were no statistically significant differences 
between RTT and non-RTT states with respect to changes between 2007-2008 and 2011-2012 in the 
number of policies and practices used, at the 0.05 level using a two-tailed test. Data from 2007-2008 
were collected retrospectively in spring 2012. All data provided by states were self-reported and not 
independently verified by the research team. For these reasons and potential concerns about recall 
accuracy, readers should exercise caution when interpreting data from 2007-2008. 


The individual policy or practice with the highest level of usage in the school 
turnaround area was providing training to the lowest-achieving schools or LEAs on 
identifying and implementing effective curricula, instructional strategies, or school 
intervention models, or developing and implementing a school improvement plan. Across 
all states, the average value for this practice was 0.93. Average values for each group of states 
(early RTT, later RTT, non-RTT) are shown in Figure IV. 19. 
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The individual policy or practice with the lowest level of usage in the school 
turnaround area was having teacher tenure rules that affect placement in or removal from 
the lowest-achieving schools. Across all states, the average value for this practice was 0.04. 
Average values for each group of states (early RTT, later RTT, non-RTT) are shown in 
Figure IV. 19. 

Figure IV.19. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on School Turnaround, Spring 2012 
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| Early RTT States (N=10-12)^Later RTT States (N=7)^ Non-RTT States (N=30-31) 


Source: Interviews with state administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each policy or 

practice in the RTT application criteria for which we identified one or more interview questions aligned 
with the policy or practice, we constructed a variable ranging from zero to one, with a value of one 
indicating that the state responded “yes” to all the interview questions selected for that policy or 
practice. The height of each bar represents the mean value of the policy or practice (on a scale of zero 
to one) for each group of states. For some of the policies and practices shown in this figure, multiple 
interview questions aligned with that policy or practice. In the figure, we indicate this using the words 
“extent to which” at the beginning of the policy or practice, to emphasize that the level of usage of that 
policy or practice is measured using multiple interview questions (as opposed to a single, binary 
measure of whether that policy or practice was used). A range is provided for the sample sizes because 
nonresponse varied across items. 

ELLs = English language learners; LEA = local education agency; PD = professional development; TA = technical 
assistance. 


F. Charter Schools 

Fostering an environment in which charter and other innovative schools can grow and thrive 
is one aim of RTT. One part of Section F of the RTT application, state reform conditions, 
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focused on ensuring successful conditions for high-perfonning charter schools and other 
innovative schools. 33 The application focused on two subtopics related to charter schools: (1) 
eliminating restrictions on charter school creation and enrollment, and (2) refining charter school 
authorization and monitoring processes. We identified four charter school policies and practices 
from the spring 2012 state interview aligned with RTT objectives on charter schools 
(Table IV.6). 

Table IV.6. Policies and Practices Aligned with RTT Objectives on Charter Schools, by Subtopic 

Eliminating Restrictions on Charter School Creation and Enrollment 

Having no restrictions on the creation of new charter schools or charter enrollment 3 

Refining Charter School Authorization and Monitoring Processes 

In considering applications for new charter schools, giving priority to schools that propose to address the needs of 
ELLs or that target ELLs 

Monitoring the academic performance of charter schools 3 
Monitoring the non-academic performance of charter schools 3 

Source: Section F of the RTT application (http://www2.ed.gov/programs/racetothetop/phase2-application.doc); 

interviews with state administrators in spring 2012. 

Note: Unless otherwise noted, the spring 2012 state interview asked only whether these policies and practices 

were in place during the current school year (2011-2012). See Appendix C for a list of the specific 
interview questions that were aligned with the RTT policies and practices in this table. See the RTT 
application for a detailed description of each subtopic. 

3 The spring 2012 state interview asked not only whether this policy or practice was in place during the current school 
year (201 1-2012) but also whether it was in place during the 2007-2008 school year. 

ELL = English language learner. 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the charter school criteria in the RTT application than 
did non-RTT states. Early RTT states reported using an average of 2.4 out of 4 policies and 
practices in this area, compared to 1.5 for non-RTT states (Figure IV. 20). For three of these four 
charter school policies and practices aligned with the RTT application criteria, no difference 
between the groups existed before the awarding of RTT grants. (Baseline data were not collected 
for the fourth policy or practice in this area, so we could not detennine if there were baseline 
differences for the area as a whole.) 

In spring 2012, early RTT states reported using statistically significantly more of the 
policies and practices aligned with the RTT application criteria than did non-RTT states 
for one charter school subtopic. Early RTT states reported using an average of 1.9 out of 3 
policies and practices in the subtopic focused on refining authorization and monitoring 
processes, compared to 1.2 for non-RTT states (Appendix A Figure A. 16). 


33 The other reform areas mentioned in Section F of the RTT application (making education funding a priority 
and demonstrating other significant reform conditions) did not focus on charter schools and are excluded from this 
analysis. 
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Figure IV. 20. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on Charter 
Schools, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 6. Each dot in this figure 

represents the number of states that reported using a particular number of policies and practices (out of four 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For example, four 
non-RTT states reported using two of the four charter school policies and practices aligned with the RTT 
application criteria. For three of the policies and practices, a “yes” response received one point. In the fourth 
case, it was possible for a state to receive a fraction of one point. See Chapter II for details on the way in 
which the number of policies and practices was determined for each state. The dashed line denotes the 
average number of policies and practices for each group of states. 

‘Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


The increases over time in the usage of charter school policies and practices aligned 
with the RTT application criteria were not statistically significantly different between RTT 
and non-RTT states. We examined the three charter school policies and practices aligned with 
the RTT application criteria for which we had state reports for both the 2007-2008 and 20 11— 
2012 school years (Figure IV. 21). The increases over time in these three policies and practices 
for early and later RTT states did not differ significantly from the increase over time for non- 
RTT states. As noted in Chapter II, these findings about changes between 2007-2008 and 20 11— 
2012 should be interpreted with caution because data from 2007-2008 were collected 
retrospectively in spring 2012. 
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Figure IV.21. Change in Reported Usage of Policies and Practices Aligned with RTT Objectives on Charter 
Schools 



— Early RTT States (N=12) — Non- RTT States (N=29-31) 

— Later RTT States (N=7) 

Source: Interviews with state administrators in spring 2012. 

Note: This figure shows change over time for each group of states in the usage of policies and practices aligned 

with the charter school section of the RTT application. The arrow for each group of states starts at the 
average number of reported policies and practices aligned with the RTT application criteria in 2007-2008 and 
ends at the average number of reported policies and practices aligned with the RTT application criteria in 
spring 2012. For example, on average, non-RTT states reported that they had used 1.3 of the 3 policies and 
practices aligned with the overall charter school topic area in 2007-2008. This group of states, on average, 
reported using 1.5 of these 3 policies and practices in spring 2012. The average usage levels for spring 2012 
differ from those reported earlier in the chapter because this figure focuses on just the subset of policies and 
practices (3 out of 4) for which we measured usage in both 2007-2008 and 2011-2012. A range is provided 
for the sample sizes because nonresponse varied across items. There were no statistically significant 
differences between RTT and non-RTT states with respect to changes between 2007-2008 and 2011-2012 in 
the number of policies and practices used, at the 0.05 level using a two-tailed test. Data from 2007-2008 
were collected retrospectively in spring 2012. All data provided by states were self-reported and not 
independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

The individual policy or practice with the highest level of usage in the charter schools 
area was monitoring the academic performance of charter schools. Across all states, the 
average value for this practice was 0.80. Average values for each group of states (early RTT, 
later RTT, non-RTT) are shown in Figure IV. 22. 

The individual policy or practice with the lowest level of usage in the charter schools 
area was giving priority to schools that propose to address the needs of ELLs or that target 
ELLs when considering applications for new charter schools. Across all states, the average 
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value for this practice was 0.02. Average values for each group of states (early RTT, later RTT, 
non-RTT) are shown in Figure IV. 22. 

Figure IV.22. States’ Reported Usage of Individual Policies and Practices Aligned with RTT 
Objectives on Charter Schools, Spring 2012 



Have No Restrictions on Prioritize Schools Monitor Academic Performance Extent to Which 

New Charter Schools or That Address Needs of ELLs of Charter Schools State Monitors 


Charter Enrollment Non-Academic 

Performance of Charter Schools 

Policy or Practice Aligned with RTT Objectives 

| Early RTT States (N=12)^Later RTT States (N=6-7) | Non-RTT States (N=30-31) 

Source: Interviews with state administrators in spring 2012. 

Note: This figure has a separate panel for each subtopic. As described in Chapter II, for each policy or 

practice in the RTT application criteria for which we identified one or more interview questions aligned 
with the policy or practice, we constructed a variable ranging from zero to one, with a value of one 
indicating that the state responded “yes” to all the interview questions selected for that policy or 
practice. The height of each bar represents the mean value of the policy or practice (on a scale of zero 
to one) for each group of states. For some of the policies and practices shown in this figure, multiple 
interview questions aligned with that policy or practice. In the figure, we indicate this using the words 
“extent to which” at the beginning of the policy or practice, to emphasize that the level of usage of that 
policy or practice is measured using multiple interview questions (as opposed to a single, binary 
measure of whether that policy or practice was used). A range is provided for the sample sizes because 
nonresponse varied across items. 

ELLs = English language learners. 

G. Summary 

This chapter examined the extent to which RTT and non-RTT states reported using RTT- 
promoted policies and practices in spring 2012 in six areas. 

On average, early RTT states reported using statistically significantly more RTT-promoted 
policies and practices than non-RTT states in five of six areas (the area of school turnaround was 
the exception), with differences ranging from 0.9 to 6.2 policies and practices per area (Figure 
IV.23). On average, later RTT states also reported using statistically significantly more RTT- 
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promoted policies and practices than non-RTT in one of six areas (the area of teacher and 
principal certification and evaluation), with a difference of 3.8 policies and practices (Figure 
IV. 23). These differences in usage of policies and practices in spring 2012 were not necessarily 
caused by RTT. In fact, among the three areas on which we collected baseline data on a subset of 
the policies and practices, we found that, for one area, a statistically significant difference 
between early RTT and non-RTT states existed prior to the RTT program in 2007-2008 (Figure 
IV. 10), and for two areas, a statistically significant difference between later RTT and non-RTT 
states existed prior to the RTT program (Figures IV. 10, IV. 18). As noted in Chapter II, the 2007- 
2008 findings should be interpreted with caution because data from 2007-2008 were collected 
retrospectively in spring 2012. 


Figure IV.23. States’ Reported Usage of Policies and Practices Promoted by RTT, by Topic Area 



(10 items) Assessments (8 items) Principal Certification Turnaround Schools 

(10 items) and Evaluation (10 items) (4 items) 

(39 items) 

| Early RTT States (N=1 2) Later RTT States (N=7)^J Non-RTT States (N=31) 

Source: Interviews with state administrators in spring 2012. 

Note: The total number of policies and practices differs by topic area. This figure reads as follows 

(using the first bar on the left as an example): early RTT states reported using 75 percent of 
the policies and practices in the state capacity area, or 7.5 out of 10 policies and practices 
examined in that area. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


Although the findings in this chapter focus on averages for the three groups of states (early 
RTT, later RTT, non-RTT), it is important to keep in mind that there was substantial variation in 
the reported usage of RTT-promoted policies and practices within each group. As a result, there 
was considerable overlap across groups in all six areas. In other words, some non-RTT states 
used more RTT-promoted policies and practices than the average number used by RTT states 
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(Figures IV. 1, IV.4, IV. 7, IV. 9, IV. 17, IV. 20); in the figures, these states are the ones in the third 
column of (green) dots that are above the dashed average lines that are shown for the first and 
second columns of (red and blue) dots. In addition, some RTT states used fewer RTT-promoted 
policies and practices than the average number used by non-RTT states (Figures IV. 1, IV.4, IV. 7, 
IV. 9, IV. 17, IV. 20); in the figures, these states are the ones in the first and second columns of 
(red and blue) dots that are below the dashed average line that is shown for the third column of 
(green) dots. 

Across all states, usage of RTT-promoted policies and practices was highest in the state 
capacity and data systems areas and lowest in the teacher and principal certification and 
evaluation area. States reported using, on average, 66 and 68 percent of RTT-promoted policies 
and practices in the state capacity and data systems areas, and 24 percent of RTT-promoted 
policies and practices in the teacher and principal certification and evaluation area (not shown). 

Across all states, the two individual policies and practices with the highest levels of usage 
were (1) having an SLDS, and (2) having a process to identify teacher shortage areas, with 
average values of 0.94 (not shown). Six of the seven individual policies and practices with the 
lowest levels of usage were concentrated in the area of teacher and principal certification and 
evaluation, and the seventh was giving priority to schools that address the needs of ELLs when 
considering applications for new charter schools; each had an average value of 0.02 (not shown). 
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Usage of Policies and Practices Promoted by RTT 


V. EXTENT TO WHICH STATES FOCUS ON ENGLISH LANGUAGE LEARNERS IN 
THEIR USAGE OF POLICIES AND PRACTICES PROMOTED BY RACE TO THE 

TOP 

English language learners (ELLs) are of particular interest to this evaluation because: (1) 
they are historically lower-achieving than non-ELLs, and (2) the RTT program placed particular 
emphasis on prioritizing the academic achievement of high-needs students, including ELLs. 
Since 2002, ELLs’ reading test scores have been below those of non-ELLs on the National 
Assessment of Educational Progress (NAEP). 34 The RTT initiative offered the opportunity to 
address this achievement gap. In particular, the RTT application criteria asked states to (1) make 
data on ELLs available to use to evaluate the effectiveness of instructional materials, strategies, 
and approaches for ELLs, and (2) provide supports and professional development to teachers and 
principals to ensure that ELLs acquire language skills to master academic content (U.S. 
Department of Education 2010). For these reasons, ED’s Office of English Language Acquisition 
(OELA) requested that part of this evaluation focus on how schools have addressed the needs of 
ELLs as they adopted the policies and practices promoted by RTT. 

In this chapter, we assess the extent to which RTT and non-RTT states reported focusing on 
ELLs in their usage of policies and practices promoted by RTT. 

We present results from four types of analyses: 

1. We compare usage of ELL-focused policies and practices aligned with the RTT 
application criteria for RTT and non-RTT states. 

2. We compare usage of these ELL-focused policies and practices for RTT and non- 
RTT states within each of the following four groups: 

a. states with higher ELL populations (defined as states with percentages of ELLs 
above the median percentage). For example, within the group of states with higher 
ELL populations, we compare early RTT states to non-RTT states; 

b. states with lower ELL populations (defined as states with percentages of ELLs 
below the median percentage); 

c. states with higher ELL/non-ELL achievement gaps (defined as states with 
achievement gaps above the median gap); and 

d. states with lower ELL/non-ELL achievement gaps (defined as states with 
achievement gaps below the median gap). 

3. Within each of the following groups of states — early RTT, later RTT, and non- 
RTT — we compare usage of ELL-focused policies and practices for states that had 
higher and lower ELL populations. For example, within the group of early RTT 
states, we compare states with higher ELL populations to states with lower ELL 
populations. 


34 National Center for Education Statistics. The Condition of Education. Accessed February 17, 2014 at 
https://nces.ed.gov/programs/coe/indicator_cgf.asp. 

35 We calculated the percentage of ELLs using CCD data on the percentage of public school students 
participating in programs for ELLs in 2007-2008. We calculated ELL/non-ELL achievement gaps as the average 
score for non-ELLs minus the average score for ELLs on the 2007 NAEP 4th grade math exam. See Chapter II for 
more details on how states were classified into higher (above-median) and lower (below-median) groups. 
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4 . Within each of the following groups of states — early RTT, later RTT, and non- 
RTT — we compare usage of ELL-focused policies and practices for states that had 
higher and lower ELL/non-ELL achievement gaps. 

Readers interested in specific examples of the individual ELL-focused policies and practices 
included in these analyses may consult Appendix D. 

Table V. 1 shows descriptive statistics on the distribution of the ELL population sizes and the 
ELL/non-ELL achievement gap for each group of states (early RTT, later RTT, non-RTT). On 
average across all states, ELLs made up 6.1 percent of the student body, and non-ELLs 
perfonned 22.5 points higher than ELLs on the NAEP exam. Early, later, and non-RTT states 
had similar ELL populations on average (6.1 percent). Later RTT states had a higher average 
ELL/non-ELL achievement gap (28.8) than early RTT states (20.2). As a result, the majority of 
later RTT states (five of seven) were classified into the group of states with higher ELL/non-ELL 
achievement gaps. 


Table V.l. Distribution of ELL Population and ELL/Non-ELL Achievement Gap 



All States 

Early RTT States Later RTT States 

Non-RTT States 

Distribution of ELL Population 

10th Percentile 

1.9 

2.7 

1.3 

1.8 

50th Percentile 

5.2 

6.5 

3.9 

5.0 

90th Percentile 

11.0 

8.9 

14.6 

11.0 

Mean 

6.1 

6.1 

6.1 

6.1 

Number of States That Had Higher and Lower ELL Populations 

Higher 

25 

7 

3 

15 

Lower 

25 

5 

4 

16 

Distribution of ELL/Non-ELL Achievement Gap 

10th Percentile 

13.0 

14.0* 

14.0 

10.0 

50th Percentile 

24.0 

22.0* 

32.5 

24.0 

90th Percentile 

32.0 

29.0* 

34.0 

30.0 

Mean 

22.5 

20.2* 

28.8 

22.2 

Number of States That Had Higher and Lower ELL/Non-ELL Achievement Gaps 

Higher 

25 a 

6 

5 

14 

Lower 

20 

6 

1 

13 

Number of States 

45-50 

12 

6-7 

27-31 


Source: Common Core of Data (CCD); National Assessment of Educational Progress (NAEP) scores. 

Note: We calculated the ELL population using CCD data on the percentage of public school 

students participating in programs for ELLs in 2007-2008. We calculated ELL/non-ELL 
achievement gaps as the average score for non-ELLs minus the average score for ELLs on 
the 2007 NAEP 4th grade math exam. States were classified into higher and lower groups 
based on whether their value (for either the ELL population or the ELL/non-ELL achievement 
gap) was above or below the median value across all states. See Chapter II for more details 
on how we classified states into groups. A range is provided for the sample size because 
missing data varied across items. There were no statistically significant differences between 
early RTT and non-RTT states, or between later RTT and non-RTT states. 
a The number of states with a higher achievement gap is not equal to the number of states with a lower 
gap because five states had the median value (and were assigned to the above-median group). 
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Table V.1 (continued) 

*Significantly different from later RTT states at the 0.05 level, two-tailed test. 

ELL = English language learner. 

We identified 12 ELL-focused policies and practices from the spring 2012 state interview 
aligned with RTT objectives (Table V.2). 


Table V.2. ELL-Focused Policies and Practices Aligned with RTT Objectives 


Prioritizing the adoption and implementation of supports to ELLs 

Providing targeted support to ELLs or working with intermediaries to provide support to ELLs 

Implementing organizational or administrative changes to improve capacity to support ELLs 

Supporting districts or schools in implementing the Common Core State Standards with ELLs 

State longitudinal data system contains program participation information about ELLs 

State longitudinal data system is used to inform and engage stakeholders and support decision makers 
in continuous improvement efforts for ELLs 

Providing funding, materials, training, technical assistance, or other supports to districts to aid in the 
use of ELL-related data 

Teacher assignment laws or policies include financial incentives to recruit and retain teachers with ELL- 
expertise 

Providing training to the lowest-achieving schools or local education agencies on identifying and 
implementing strategies to address the need of ELLs 

Having state-level staff or consultants to support turnaround schools and districts in working with ELLs 

In considering applications for new charter schools, giving priority to schools that propose to address 
the needs of ELLs or that target ELLs 

Monitoring charter school performance based on the student populations (such as ELLs) served 


Source: RTT application (http://www2.ed.gov/programs/racetothetop/phase2-application.doc); 

interviews with state administrators in spring 2012. 

Note: See Appendix D for a list of the specific interview questions that were aligned with the ELL- 

focused policies and practices in this table. All the policies and practices listed in this table 
were included in the main analyses described in Chapter IV, but some of them are not listed 
in the Chapter IV tables because they were included in a broader policy or practice that is 
listed in those tables. 

ELL = English language learner. 

In spring 2012, there was no statistically significant difference in usage of ELL-focused 
policies and practices aligned with the RTT application criteria between RTT and non- 
RTT states. Early RTT states, later RTT states, and non-RTT states reported using an average of 
6.3, 6.1, and 5.6 of 12 ELL-focused policies and practices aligned with the RTT application 
criteria (Figure V.l). 

Within each of the following four groups — states with higher ELL populations, states 
with lower ELL populations, states with higher ELL/non-ELL achievement gaps, and 
states with lower ELL/non-ELL achievement gaps — there were no statistically significant 
differences between RTT and non-RTT states in the number of ELL-focused policies and 
practices used. Early RTT states, later RTT states, and non-RTT states with higher ELL 
populations reported using an average of 7.2, 6.7, and 6.4 of 12 ELL-focused policies and 
practices aligned with the RTT application criteria, while states with lower ELL populations 
reported using 5.0, 5.8, and 4.8 ELL-focused policies and practices (Figure V.2). Early RTT 
states, later RTT states, and non-RTT states with higher ELL/non-ELL achievement gaps 
reported using an average of 5.7, 6.8, and 5.8 ELL-focused policies and practices, while states 
with lower ELL/non-ELL achievement gaps reported using 6.8, 6.3, and 5.7 ELL-focused 
policies and practices (Figure V.3). 
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Figure V.l. States’ Reported Usage of ELL-Focused Policies and Practices Aligned with RTT 
Objectives, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table V.2. Each dot in 

this figure represents the number of states that reported using a particular number of ELL- 
focused policies and practices (out of 12 examined) that were aligned with the RTT 
application criteria. The number inside each dot is the number of states represented by the 
dot; dots that represent only one state have no number inside. For example, three non-RTT 
states reported using 6 of 12 ELL-focused policies and practices aligned with the RTT 
application criteria. For 6 of the policies and practices, a “yes” response received one point. 
In the other 6 cases, it was possible for a state to receive a fraction of one point. See Chapter 
II for details on the way in which the number of ELL-focused policies and practices was 
determined for each state. The dashed line denotes the average number of ELL-focused 
policies and practices for each group of states. There were no statistically significant 
differences between RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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Figure V.2. States’ Reported Usage of ELL-Focused Policies and Practices Aligned with RTT 
Objectives, by ELL Population, Spring 2012 



ELL Population 

Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table V.2. Each column in the figure 

shows the number of ELL-focused policies and practices that states in each group (early RTT, later RTT, non- 
RTT) reported using, by states that had higher and lower ELL populations. Each dot in this figure represents 
the number of states that reported using a particular number of ELL-focused policies and practices (out of 12 
examined) that were aligned with the RTT application criteria. The number inside each dot is the number of 
states represented by the dot; dots that represent only one state have no number inside. For example, two 
non-RTT states that had higher ELL populations reported using 6 of 12 ELL-focused policies and practices 
aligned with the RTT application criteria. For 6 of the policies and practices, a “yes” response received one 
point. In the other 6 cases, it was possible for a state to receive a fraction of one point. See Chapter II for 
details on the way in which the number of ELL-focused policies and practices was determined for each state. 
The dashed line denotes the average number of ELL-focused policies and practices for each group of states. 
Within each of the following two groups — states with higher ELL populations and states with lower ELL 
populations — none of the differences in the average number of ELL-focused policies and practices used by 
early RTT, later RTT, and non-RTT states were statistically significant at the 0.05 level using a two-tailed test. 

^Significantly different from states within the same group that had lower ELL populations at the 0.05 level, two-tailed test. 

For example, early RTT states with higher ELL populations used statistically significantly more of the ELL-focused policies 

and practices aligned with the RTT application criteria than did early RTT states with lower ELL populations. 
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Figure V.3. States’ Reported Usage of ELL-Focused Policies and Practices Aligned with RTT 
Objectives, by ELL/Non-ELL Achievement Gap, Spring 2012 



ELL/Non-ELL Achievement Gap 

Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table V.2. Each column in the figure 

shows the number of ELL-focused policies and practices that states in each group (early RTT, later RTT, non- 
RTT) reported using, by states that had higher and lower achievement gaps between ELLs and non-ELLs. 
Each dot in this figure represents the number of states that reported using a particular number of ELL-focused 
policies and practices (out of 12 examined) that were aligned with the RTT application criteria. The number 
inside each dot is the number of states represented by the dot; dots that represent only one state have no 
number inside. For example, two non-RTT states that had a higher ELL/non-ELL achievement gap reported 
using 6 of 12 ELL-focused policies and practices aligned with the RTT application criteria. For 6 of the ELL- 
focused policies and practices, a “yes” response received one point. In the other 6 cases, it was possible for a 
state to receive a fraction of one point. See Chapter II for details on the way in which the number of ELL- 
focused policies and practices was determined for each state. The dashed line denotes the average number 
of ELL-focused policies and practices for each group of states. There were no statistically significant 
differences at the 0.05 level using a two-tailed test (1) between early RTT, later RTT, and non-RTT states with 
the same ELL/non-ELL achievement gap classification, and (2) between higher and lower ELL/non-ELL 
achievement gap states with the same RTT status. 
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Within each group of states — early RTT, later RTT, and non-RTT — the states with 
higher ELL populations used statistically significantly more of the ELL-focused policies 
and practices aligned with the RTT application criteria than states with lower ELL 
populations. Early RTT states with higher ELL populations reported using 7.2 of 12 ELL- 
focused policies and practices, compared with 5.0 for early RTT states with lower ELL 
populations (Ligure V.2). Later RTT states with higher ELL populations reported using 6.7 of 12 
ELL-focused policies and practices, compared with 5.8 for later RTT states with lower ELL 
populations. Non-RTT states with higher ELL populations reported using 6.4 of 12 ELL-focused 
policies and practices, compared with 4.8 for non-RTT states with lower ELL populations. 

Within each group of states — early RTT, later RTT, and non-RTT — there was no 
statistically significant difference in usage of the ELL-focused policies and practices aligned 
with the RTT application criteria between states with higher and lower ELL/non-ELL 
achievement gaps. Early RTT, later RTT, and non-RTT states with higher ELL/non-ELL 
achievement gaps reported using 5.7, 6.8, and 5.8 of 12 ELL-focused policies and practices, 
while those states with lower ELL/non-ELL achievement gaps reported using 6.8, 6.3, and 5.7 
policies and practices (Ligure V.3). 

The individual ELL-focused policies or practices with the highest levels of usage were 
(1) providing targeted support to ELLs or working with intermediaries to provide support 
to ELLs, and (2) providing training to the lowest-achieving schools or local education 
agencies on identifying and implementing strategies to address the need of ELLs. Across all 
states, the average values for these policies and practices were 0.78 and 0.76. Average values for 
each group of states (early RTT, later RTT, non-RTT) are shown in Ligure V.4. 

The individual ELL-focused policies or practices with the lowest levels of usage were 
(1) having teacher assignment laws or policies that include financial incentives to recruit 
and retain teachers with ELL expertise, and (2) giving priority to schools that propose to 
address the needs of ELLs or that target ELLs when considering applications for new 
charter schools. Across all states, the average values for these policies and practices were 0.00 
and 0.02. Average values for each group of states (early RTT, later RTT, non-RTT) are shown in 
Ligure V.5. 
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Figure V.4. States’ Reported Usage of the First Six Individual ELL-Focused Policies and Practices 
Aligned with RTT Objectives, Spring 2012 



Extent to Extent to Extent to Support CCSS Extent to Extent to 

Which State Which State Which State Implementation Which SLDS Which State 

Prioritized Provides Implements for ELLs Includes Program Uses SLDS 

Supports Targeted Support Organizational Participation to Support 

to ELLs to ELLs Changes to about ELLs ELL-Related 

Improve Capacity to Decisions 

Support ELLs 

ELL-Focused Policy or Practice Aligned with RTT Objectives 


| Early RTT States (N=12) J Later RTT States (N=7)^Non-RTT States (N=30-31) 


Source: Interviews with state administrators in spring 2012. 

Notes: As described in Chapter II, for each ELL-focused policy or practice aligned with the RTT application 

criteria for which we identified one or more interview questions aligned with the policy or practice, we 
constructed a variable ranging from zero to one, with a value of one indicating that the state responded 
“yes” to all the interview questions selected for that policy or practice. The height of each bar represents 
the mean value of the ELL-focused policy or practice (on a scale of zero to one) for each group of 
states. For some of the policies and practices shown in this figure, multiple interview questions aligned 
with that policy or practice. In the figure, we indicate this using the words “extent to which” at the 
beginning of the policy or practice, to emphasize that the level of usage of that policy or practice is 
measured using multiple interview questions (as opposed to a single, binary measure of whether that 
policy or practice was used). 

ELL = English language learner; CCSS = Common Core State Standards; SLDS = State longitudinal data system. 
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Figure V.5. States’ Reported Usage of the Last Six Individual ELL-Focused Policies and Practices 
Aligned with RTT Objectives, Spring 2012 
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Source: Interviews with state administrators in spring 2012. 

Notes: As described in Chapter II, for each ELL-focused policy or practice aligned with the RTT application 

criteria for which we identified one or more interview questions aligned with the policy or practice, we 
constructed a variable ranging from zero to one, with a value of one indicating that the state responded 
“yes” to all the interview questions selected for that policy or practice. The height of each bar represents 
the mean value of the ELL-focused policy or practice (on a scale of zero to one) for each group of 
states. For some of the policies and practices shown in this figure, multiple interview questions aligned 
with that policy or practice. In the figure, we indicate this using the words “extent to which” at the 
beginning of the policy or practice, to emphasize that the level of usage of that policy or practice is 
measured using multiple interview questions (as opposed to a single, binary measure of whether that 
policy or practice was used). 

ELL = English language learner. 
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VI. DISCUSSION OF MAIN FINDINGS FOR RACE TO THE TOP 

As noted in Chapter I, few studies on the implementation of RTT-promoted policies and 
practices examine a comprehensive set of these policies and practices spanning all the topic areas 
described in the RTT application, and few examine whether the policies and practices used by 
RTT states differ from those used by non-RTT states. This volume of the report seeks to address 
these gaps in the existing literature by examining the extent to which RTT and non-RTT states 
reported using RTT-promoted policies and practices in six areas: (1) state capacity, (2) standards 
and assessments, (3) data systems, (4) teacher and principal certification and evaluation, (5) 
school turnaround, and (6) charter schools. We also assessed the extent to which states reported 
focusing on English language learners (ELLs) in their usage of policies and practices promoted 
by RTT. The analysis compared the 12 early RTT states with the 31 non-RTT states and the 7 
later RTT states with the 31 non-RTT states. We focused on policies and practices that states 
reported using in spring 2012 but also collected information on certain policies and practices 
states reported having in place during the 2007-2008 school year, before the RTT program was 
announced. 

Early RTT states reported using statistically significantly more of the policies and practices 
promoted by RTT, on average, than non-RTT states in five of the six areas considered (school 
turnaround is the exception). This finding is consistent with two previous studies of RTT 
implementation that suggested that RTT states were more likely to adopt RTT-promoted policies 
and practices than other states (Government Accountability Office 2011, National Council on 
Teacher Quality 2011). The difference between early RTT and non-RTT states in the average 
number of RTT-promoted policies and practices used ranged from 0.9 to 6.2 across the five 
areas. However, some of these differences predated award of the RTT grants. In particular, early 
RTT states reported having already used statistically significantly more of some teacher and 
principal certification and evaluation policies and practices promoted by RTT than non-RTT 
states before receiving their RTT grants. As noted in Chapter II, the 2007-2008 findings should 
be interpreted with caution because data from 2007-2008 were collected retrospectively in 
spring 2012. For several areas (state capacity, standards and assessments, data systems), we do 
not have information on states’ reported usage of RTT-promoted policies and practices prior to 
the awarding of RTT grants; therefore, for those areas, we cannot detennine if some of the 
differences between early RTT states and non-RTT states in spring 2012 predated receipt of the 
RTT grants. 

Perhaps not surprising given the limited time (about four months) between when grants were 
awarded to later RTT states and our spring 2012 interviews, we found fewer differences between 
later RTT states and non-RTT states. The average number of RTT-promoted policies and 
practices used by later RTT states was statistically significantly larger than that for non-RTT 
states in one of six areas (the area of teacher and principal certification and evaluation), with a 
difference of 3.8 policies and practices. As with the early RTT states, there was some evidence 
of differences for the teacher and principal certification and evaluation area between later RTT 
and non-RTT states before RTT grants were awarded. 

Although the findings reported above focus on averages for each group of states (early RTT, 
later RTT, non-RTT), there was substantial variation in the reported usage levels of RTT- 
promoted policies and practices within each group. 

Usage of policies and practices promoted by RTT was highest in the state capacity and data 
systems areas and lowest in the teacher and principal certification and evaluation area. States 
reported using, on average, 66 and 68 percent of RTT-promoted policies and practices in the 

66 



VI. Discussion of Main Findings for Race to the Top 


Usage of Policies and Practices Promoted by RTT 


state capacity and data systems areas and 24 percent of RTT-promoted policies and practices in 
the teacher and principal certification and evaluation area. 

Across all states, the two individual policies and practices with the highest levels of usage 
were (1) having an SLDS, and (2) having a process to identify teacher shortage areas. Six of the 
seven individual policies and practices with the lowest levels of usage were concentrated in the 
area of teacher and principal certification and evaluation, and the seventh was giving priority to 
schools that address the needs of ELLs when considering applications for new charter schools. 

States reported that their usage of RTT-promoted policies and practices increased over time 
in three areas — (1) teacher and principal certification and evaluation, (2) school turnaround, and 
(3) charter schools — but these increases were not statistically significantly different between 
RTT and non-RTT states. For each area, on average across states, the increase between 2007- 
2008 and 2011-2012 was between one and two policies and practices. For the other three areas 
(state capacity, standards and assessments, data systems), we do not have information on states’ 
reported usage of RTT-promoted policies and practices prior to spring 2012, so we cannot 
detennine whether usage increased over time. 

The average number of ELL-focused policies and practices promoted by RTT that early and 
later RTT states reported using was not statistically significantly different. Within each group of 
states (early RTT, later RTT, non-RTT), states with higher ELL populations used statistically 
significantly more ELL-focused policies and practices than states with lower ELL populations, 
but there were no statistically significant differences in usage between states with higher and 
lower ELL/non-ELL achievement gaps. 

Readers may have questions about potential explanations for the observed pattern of 
findings. Below, we lay out several questions of possible interest and potential explanations for 
these findings. 

Why did early RTT states report using more policies and practices aligned with the 
RTT application criteria than non-RTT states? We focus here on two potential explanations. 
First, this finding could reflect the effect of the RTT grants on state policies and practices. RTT 
grants aimed to increase usage of the policies and practices examined, so this pattern of findings 
could reflect progress toward that goal. Second, this finding could reflect differences in the usage 
of these policies and practices between the groups of states prior to the awarding of the RTT 
grants. For the three areas for which we could examine the prevalence of RTT-promoted policies 
and practices both before and after RTT grants were awarded, evidence suggests that some 
differences existed prior to the RTT awards. Further, the RTT application criteria gave explicit 
priority to states that had already used policies consistent with the RTT goals: “over half the 
points that reviewers may award to States are based on States’ accomplishments prior to 
applying” (Race to the Top Application for Phase 2 Funding, p. 76). Therefore, it is possible that 
the higher usage levels of RTT-promoted practices observed among early RTT states are at least 
partly an artifact of the selection process rather than a result of the RTT grant awards, but this 
does not mean that RTT had no effect on state policies and practices. Rather, RTT may have 
affected policies and practices in many states simply by holding the competition; in other words, 
many states (including both those that eventually received RTT awards and those that didn’t) 
may have increased their usage of RTT-promoted policies and practices after 2007-2008 to 
better position themselves to receive an RTT award in 2010 or 201 1. Our finding that usage of 
RTT-promoted policies and practices increased between 2007-2008 and 2011-2012 for all states 
(and not just RTT states) in three areas is consistent with this hypothesis. 
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Why did early RTT states not report using more RTT-promoted school turnaround 
policies and practices than non-RTT states? We focus here on two potential explanations for 
this finding. First, it is possible that RTT did not have an effect on practices in this area among 
states. Second, it is possible that usage of school turnaround practices is on the rise more broadly 
(even in states that did not receive RTT grants), so that we did not observe the RTT program 
resulting in higher levels of usage among RTT states relative to non-RTT states. School 
turnaround practices may be rising nationally, in part, due to the School Improvement Grant 
funds that all states receive, of which 5 percent can be set aside to support activities associated 
with school turnaround. Non-RTT states reported using more policies and practices in the school 
turnaround area in 2012 than before RTT, suggesting greater usage of turnaround practices even 
in states that did not receive RTT grants. 

Why did later RTT states not report using more RTT-promoted policies and practices 
than non-RTT states? Few of the differences between later RTT states and non-RTT states 
were statistically significant. There are several potential explanations for this finding. First, it 
may reflect the little time that had elapsed between receipt of the RTT awards by the later RTT 
states and our spring 2012 interviews. Second, later grants were smaller and more limited in 
focus. Later RTT states generally focused on a subset of the six areas addressed by early RTT 
states. All seven later RTT states focused on the areas of state capacity and standards and 
assessments, six focused on data systems and teacher and principal certification and evaluation, 
two focused on charter schools, and one focused on school turnaround. This more limited focus 
might make it less likely for differences between later RTT states and non-RTT states to exist for 
any given area. However, our finding that later RTT states reported using statistically 
significantly more RTT-promoted teacher and principal certification and evaluation practices 
than did non-RTT states is consistent with the fact that six of the seven later RTT states focused 
on this area. 

Third, our findings may also reflect similarities between these two groups of states prior to 
the awarding of the RTT grants. We found no statistically significant differences between later 
RTT states and non-RTT states in the usage of RTT-promoted policies and practices prior to 
grant award for one area (charter schools). However, there were preexisting differences between 
later RTT states and non-RTT states for two other areas (teacher and principal certification and 
evaluation, school turnaround). (We do not have pre-award information for the remaining three 
areas.) 

Fourth, since the later RTT states did not receive grants in the first or second rounds of 
competition (despite having applied in earlier rounds), it is also possible that our findings reflect 
limitations in these states’ capacity for policy changes. Finally, our statistical power (that is, our 
ability to detect differences between the groups of states) was lower for the comparisons of later 
RTT and non-RTT states than for the comparisons of early RTT and non-RTT states, due in part 
to the smaller sample size. Therefore, even if there were differences between later RTT and non- 
RTT states of a similar size as the significant differences we found between early RTT and non- 
RTT states, they might not have been statistically significant. However, while later RTT states 
reported using more policies and practices than non-RTT states in all areas, these differences 
were smaller than the differences between early RTT and non-RTT states. 

Why was usage of RTT-promoted policies and practices higher in some areas than in 
others? We focus here on two possible explanations. First, certain RTT-promoted policies and 
practices may require more time to adopt, or have more barriers to usage, than others. For 
example, a 2013 GAO report on RTT found that “officials in most states cited challenges related 
to developing and using evaluation measures [and] addressing teacher concerns” (Government 
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Accountability Office 2013), and a 2013 study by Weiss focused on the difficulty states have had 
implementing new teacher and principal evaluation systems. Our finding that usage of RTT- 
promoted policies and practices was lowest in the teacher and principal certification and 
evaluation area is consistent with this hypothesis. Our finding that usage of RTT-promoted 
policies and procedures was highest in the area of state capacity may be reassuring to RTT 
observers, given that prior studies have raised concerns about lack of capacity (Boser 2012). 
Second, it is possible that RTT states purposefully used certain policies before others, and plan to 
eventually use more policies. Early RTT states still had substantial proportions of their grant 
money left to spend as of November 2013, so it is possible that these states are still planning to 
use more policies and practices in the areas for which we found the lowest usage (McNeil 2013). 

Why did RTT states not report using more ELL-focused policies and practices than 
non-RTT states? Again, we focus here on two possible explanations. First, it is possible that 
RTT did not have an effect on ELL-focused policies and practices among states because states 
are focused on policies and practices that are thought to be effective for all students, including 
ELLs, rather than policies and practices explicitly focused on ELLs. Second, it is possible that 
ELL-focused policies and practices are on the rise primarily in states with higher ELL 
populations (which include some RTT states and some non-RTT states). Our finding that within 
each group of states (early RTT, later RTT, non-RTT), states with higher ELL populations 
reported using statistically significantly more ELL-focused policies and practices than states with 
lower ELL populations is consistent with this hypothesis. 

Although we cannot definitively accept or reject any of these possible explanations for these 
findings, we offer them as starting points for future investigations into the implementation of 
RTT-promoted policies and practices. Because the process of changing education policies and 
practices can be complex and require substantial time to implement, we administered a second 
round of state interviews in spring 2013 so that we can continue to explore progress toward 
usage of RTT policies and practices after another year of the program. We will revisit these 
patterns and findings in a future report. 
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This appendix contains additional figures that are directly related to the analyses presented 
in Chapter IV. In this appendix, we focus on the same six areas addressed in Chapter IV: (1) state 
capacity, (2) standards and assessments, (3) data systems, (4) teacher and principal certification 
and evaluation, (5) school turnaround, and (6) charter schools. For each area, we present a series 
of figures, one for each subtopic, showing states’ reported usage of the policies and practices 
aligned with the Race to the Top (RTT) application criteria for that subtopic, similar to the 
figures shown in Chapter IV for each topic area. 

A. State Capacity 

Figure A.1. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on State 
Capacity, Articulating State’s Reform Agenda Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 1. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of five examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For two of the policies and practices, a “y es ” response 
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Figure A.1 (continued) 

received one point. In the other three cases, it was possible for a state to receive a fraction of 
one point. See Chapter II for details on the way in which the number of policies and practices 
was determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 

Figure A.2. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on State 
Capacity, Building Strong Statewide Capacity Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 1. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of five examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For one policy or practice, a “yes” response received 
one point. In the other four cases, it was possible for a state to receive a fraction of one point. 
See Chapter II for details on the way in which the number of policies and practices was 
determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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B. Standards and Assessments 

Figure A.3. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Standards and Assessments, Developing and Adopting Common Standards Subtopic, Spring 
2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 2. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of one examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For this policy or practice, a “yes” response received 
one point. See Chapter II for details on the way in which the number of policies and practices 
was determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. There were no statistically significant differences between 
RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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Figure A.4. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Standards and Assessments, Developing and Implementing Common, High Quality Assessments 
Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 2. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of two examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For these policies and practices, a “yes” response 
received one point. See Chapter II for details on the way in which the number of policies and 
practices was determined for each state. The dashed line denotes the average number of 
policies and practices for each group of states. There were no statistically significant 
differences between RTT and non-RTT states at the 0.05 level using a two-tailed test. 


A.5 


Appendix A. Additional Figures Based on State Interviews 


Usage of Policies and Practices Promoted by RTT 


Figure A.5. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Standards and Assessments, Supporting the Transition to Enhanced Standards and High Quality 
Assessments Subtopic, Spring 2012 

o 

Q. 

O 



Note: The policies and practices summarized in this figure are presented in Table IV.2. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of seven examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For three of policies and practices, a “yes” response 
received one point. In the other four cases, it was possible for a state to receive a fraction of 
one point. See Chapter II for details on the way in which the number of policies and practices 
was determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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C. Data Systems 

Figure A.6. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on Data 
Systems, Fully Implementing a SLDS Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 3. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of four examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For these policies and practices, a “yes” response 
received one point. See Chapter II for details on the way in which the number of policies and 
practices was determined for each state. The dashed line denotes the average number of 
policies and practices for each group of states. There were no statistically significant 
differences between RTT and non-RTT states at the 0.05 level using a two-tailed test. 

SLDS = state longitudinal data system. 
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Figure A.7. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on Data 
Systems, Accessing State Data and Using It to Inform Key Stakeholders Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 3. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of two examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For these policies and practices, a “yes” response 
received a fraction of one point. See Chapter II for details on the way in which the number of 
policies and practices was determined for each state. The dashed line denotes the average 
number of policies and practices for each group of states. There were no statistically 
significant differences between RTT and non-RTT states at the 0.05 level using a two-tailed 
test. 
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Figure A.8. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on Data 
Systems, Using Data to Improve Instruction Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 3. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of two examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For these policies and practices, a “yes” response 
received one point. See Chapter II for details on the way in which the number of policies and 
practices was determined for each state. The dashed line denotes the average number of 
policies and practices for each group of states. There were no statistically significant 
differences between RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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D. Teacher and Principal Certification and Evaluation 

Figure A.9. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Teacher and Principal Certification and Evaluation, Providing High Quality Pathways to 
Certification for Aspiring Teachers and Principals Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 4. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of 14 examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For 6 of the policies and practices, a “yes” response 
received one point. In the other 8 cases, it was possible for a state to receive a fraction of one 
point. See Chapter II for details on the way in which the number of policies and practices was 
determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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Figure A.10. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Teacher and Principal Certification and Evaluation, Improving Teacher and Principal Effectiveness 
Based on Performance Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV.4. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of 15 examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For 11 of the policies and practices, a “yes” response 
received one point. In the other 4 cases, it was possible for a state to receive a fraction of one 
point. See Chapter II for details on the way in which the number of policies and practices was 
determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 


A.l 1 


Appendix A. Additional Figures Based on State Interviews 


Usage of Policies and Practices Promoted by RTT 


Figure A.11. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Teacher and Principal Certification and Evaluation, Ensuring Equitable Distribution of Effective 
Teachers and Principals Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 4. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of four examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For two of the policies and practices, a “yes” response 
received one point. In the other two cases, it was possible for a state to receive a fraction of 
one point. See Chapter II for details on the way in which the number of policies and practices 
was determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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Figure A.12. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Teacher and Principal Certification and Evaluation, Improving the Effectiveness of Teacher and 
Principal Preparation Programs Subtopic, Spring 2012 

o 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 4. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of six examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For these policies and practices, a “yes” response 
received one point. See Chapter II for details on the way in which the number of policies and 
practices was determined for each state. The dashed line denotes the average number of 
policies and practices for each group of states. There were no statistically significant 
differences between RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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E. School Turnaround 

Figure A.13. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
School Turnaround, Intervening in the Lowest-Achieving Schools and LEAs Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policy or practice summarized in this figure is presented in Table IV. 5. Each dot in this 

figure represents the states that reported using the one policy or practice that was aligned 
with the criterion in the RTT application about intervening in the lowest-achieving schools. 
The number inside each dot is the number of states represented by the dot; dots that 
represent only one state have no number inside. See Chapter II for details on the way in 
which the number of policies and practices was determined for each state. The dashed line 
denotes the average number of policies and practices for each group of states. There were 
no statistically significant differences between RTT and non-RTT states at the 0.05 level 
using a two-tailed test. 

LEA = local education agency. 
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Figure A.14. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
School Turnaround, Turning Around the Lowest-Achieving Schools Subtopic, Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 5. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of nine examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For five of the policies and practices, a “yes” response 
received one point. In the other cases, it was possible for a state to receive a fraction of one 
point. See Chapter II for details on the way in which the number of policies and practices was 
determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. There were no statistically significant differences between 
RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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F. Charter Schools 

Figure A.15. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Charter Schools, Eliminating Restrictions on Charter School Creation and Enrollment Subtopic, 
Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policy or practice summarized in this figure is presented in Table IV. 6. Each dot in this 

figure represents the states that reported using the one policy or practice that was aligned 
with the RTT application criteria. The number inside each dot is the number of states 
represented by the dot; dots that represent only one state have no number inside. See 
Chapter II for details on the way in which the number of policies and practices was 
determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. There were no statistically significant differences between 
RTT and non-RTT states at the 0.05 level using a two-tailed test. 
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Figure A.16. States’ Reported Usage of Policies and Practices Aligned with RTT Objectives on 
Charter Schools, Refining Authorization and Monitoring Processes for Charter Schools Subtopic, 
Spring 2012 



Source: Interviews with state administrators in spring 2012. 

Note: The policies and practices summarized in this figure are presented in Table IV. 6. Each dot in 

this figure represents the states that reported using a particular number of policies and 
practices (out of three examined) that were aligned with the RTT application criteria. The 
number inside each dot is the number of states represented by the dot; dots that represent 
only one state have no number inside. For two of the policies and practices, a “yes” response 
received one point. In the third case, it was possible for a state to receive a fraction of one 
point. See Chapter II for details on the way in which the number of policies and practices was 
determined for each state. The dashed line denotes the average number of policies and 
practices for each group of states. 

*Significantly different from non-RTT states at the 0.05 level, two-tailed test. 
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In the main body of the report, we summarized the extent to which states reported using the 
policies and practices promoted by RTT, including the individual policies and practices within 
each topic area and subtopic. As described in Chapter II, some policies or practices were 
comprised of multiple interview questions. In this appendix, we present detailed findings from 
particular individual interview questions and describe how we analyzed those data. Specifically, 
we show the number of early RTT states, later RTT states, and non-RTT states that responded 
“yes” to each interview question examined as part of this report. Readers interested in states’ 
responses to individual interview questions may find this appendix useful. 

In Section A, we discuss how we analyzed data from closed- and open-ended questions and 
how we handled missing values. In Section B, we present findings from the interview questions 
in a series of tables, the titles of which are shown in the list of tables at the beginning of this 
report. Two of the tables in Section B involve data from sources other than the state interviews 
described in Chapter II: in Table B.48, we present data from the Common Core of Data; and in 
Table B.59, we present data from the National Alliance for Public Charter Schools. 

A. Analysis Methods 

Analyzing data from closed-ended questions. The evaluation’s protocol for interviews 
with state administrators comprised mostly closed-ended questions — that is, questions with “yes” 
or “no” responses or with a set of specific response categories from which to choose. As a result, 
these variables were already in a format that is suitable, or nearly suitable, for analysis. 

Closed-ended questions sometimes included an “other-specify” response option so the 
interview could progress smoothly when a respondent was uncertain about the response option 
that applied or could not find a response option that adequately captured the response he or she 
wished to provide. When a respondent chose this option, the interviewer asked the respondent to 
specify his or her response and recorded it. These “other-specify” responses were reviewed and 
either recoded into one of the existing structured response categories or coded into a new 
response category, as appropriate. Following reporting requirements established by the U.S. 
Department of Education’s National Center for Education Statistics, we created a new response 
category only if at least three respondents (that is, states) provided the same or similar response. 
If fewer than three respondents provided a particular response, the response remained part of the 
broad “other” category. 

Analyzing data from open-ended questions. Whenever possible, we categorized the 
responses to open-ended questions into nominal categories (based on the themes that emerged) 
that could then be treated as quantitative, categorical data. This strategy enabled us to 
systematically identify and report on recurring themes mentioned frequently by respondents. 

Handling missing values. Values can be missing for various reasons: (1) because the 
respondent did not complete the interview; (2) because the respondent completed the interview 
but did not complete the question; (3) because the respondent chose “don’t know,” “refused,” or 
“not applicable”; or (4) because the question was logically skipped based on earlier responses. 
Generally, we excluded all missing values from our calculations regardless of the reason that the 
question was missing (that is, we did not recode a missing as a zero). In the tables, we report the 
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sample sizes for states with nonmissing values on the given item. Frequencies generally total 50 
(the number of states that completed the interview). 1 

Selecting interview questions aligned with the RTT application criteria. As described in 
Chapter II, we reviewed the interview questions and assigned those that aligned with the policies 
and practices described in the RTT application selection criteria to specific topic areas and 
subtopics. We determined the subtopic into which each interview question fell based on the 
section of the RTT application criteria with which it aligned. In the tables presented in Section B, 
the last column of each table indicates whether each question was selected, and if it was selected, 
for which subtopic, by using the abbreviations shown in Table B.l. 


Table B.l. Abbreviations for Subtopics 

Subtopic Abbreviation 

Topic Area: State Capacity 

Articulating the state’s education reform agenda and LEAs' participation in it SC-1 

Building strong statewide capacity to implement, scale up, and sustain the 
proposed plans SC-2 

Topic Area: Standards and Assessments 

Developing and adopting common standards SA-1 

Developing and implementing common, high-quality assessments SA-2 

Supporting the transition to enhanced standards and high-quality assessments SA-3 

Topic Area: Data Systems and Use of Data to Improve Instruction 
Fully implementing a statewide longitudinal data system DA-1 

Accessing state data and using it to inform key stakeholders DA-2 

Using data to improve instruction DA-3 

Topic Area: Teacher and Principal Certification and Evaluation 
Providing high-quality pathways to certification for aspiring teachers and 
principals TL-1 

Improving teacher and principal effectiveness based on performance TL-2 

Ensuring equitable distribution of effective teachers and principals TL-3 

Improving the effectiveness of teacher and principal preparation programs TL-4 

Topic Area: School Turnaround 

Authority to intervene in the lowest-achieving schools and LEAs TA-1 

Turning around the lowest-achieving schools TA-2 

Topic Area: Charter Schools 

Eliminating restrictions on charter school creation and enrollment CH-1 

Refining authorization and monitoring processes for charter schools CH-2 

Source: Interviews with state administrators in spring 2012. 

LEAs = local education agencies. 


1 In some cases, the number of states in a table totals more than 50; we include a footnote to those tables 
explaining why. As one example, if the question asked the respondent to mark all responses that apply, respondents 
could choose multiple answers. 
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B. Detailed Findings from Interview Questions 

In this section we present findings from particular interview questions based on U.S. 
Department of Education guidance about those in which it had the most interest. The tables are 
organized to follow the order of the modules in the state interview protocol, which was: (1) state 
capacity, (2) standards and assessments, (3) data systems, (4) teachers and leaders, (5) school 
turnaround, and (6) charter schools. 


Table B.2. State Implementation of Comprehensive Education Reform Plans, Spring 2012 


- 

Early RTT 

Number of States 
Later RTT 

Non-RTT 

Item Aligned 
with RTT 
Application 
Criteria 
(Subtopic) 

Reported having a comprehensive 
education reform plan in place 

12 

7 

27 

Yes (SC-1) 

Reported using the following 
strategies to implement this reform 
plan: 

Working with districts that have 
the lowest achievement levels 
to improve their performance 

12 

7 

26 

No 

Working with districts that have 
the highest achievement levels 
and seeking to replicate their 
practices statewide 

11 

4 

12 

Yes (SC-2) 

Working with a specific set of 
districts that are not 
necessarily the highest or 
lowest achieving in the state 

12 

6 

20 

No 

Other strategy 

3 

3 

12 

No 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

SC-1 = Articulating the State’s education reform agenda and LEAs' participation in it; SC-2 = Building 
strong statewide capacity to implement, scale up, and sustain the proposed plans. 
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Table B.3. States’ Education Reform Priorities, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported prioritizing the following 
education reform initiatives to a 
great extent: 

Adopting and implementing 
college- and career-ready 
standards and/or assessments 

10 

7 

29 

Yes (SC-1) 

Building comprehensive, student- 
level, longitudinal data systems 

10 

5 

24 

Yes (SC-1) 

Using data to improve instruction 

12 

6 

23 

Yes (SC-1) 

Recruiting and/or retaining 
effective teachers and school 
leaders 

7 

2 

15 

Yes (SC-1) 

Rewarding effective teachers 
and school leaders 

3 

1 

6 

Yes (SC-1) 

Developing and preparing 
effective teachers and school 
leaders 

9 

4 

19 

Yes (SC-1) 

Developing and implementing a 
teacher and principal 
evaluation system that is 
based on student growth 

11 

7 

24 

Yes (SC-1) 

Improving the distribution of 
effective teachers and 
principals 

5 

2 

6 

Yes (SC-1) 

Turning around the lowest- 
achieving schools 

12 

6 

24 

Yes (SC-1) 

Providing supports for ELLs 

4 

2 

14 

No 

Reported that ELLs fit into their 
current education reform priorities 
in the following ways: 

ELLs are an explicit, central 
priority of statewide reform 
efforts 

1 

0 

5 

No 

ELLs are an emerging priority of 
statewide reform efforts 

2 

1 

8 

No 

Statewide reform efforts are 
designed to address the needs 
of all students, including ELLs 

9 

6 

17 

No 

Number of States 

11-12 

7 

30-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

ELLs = English language learners; SC-1 = Articulating the State’s education reform agenda and LEAs' 
participation in it. 
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Table B.4. Role of State Education Agency in Education Reform, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported that the SEA played the 
following roles to a great extent: 
Creator of a statewide vision for 
reforms 

11 

7 

24 

Yes (SC-2) 

Creator and monitor of 
performance measures 

11 

6 

28 

Yes (SC-2) 

Compliance monitor of reform 

11 

7 

28 

Yes (SC-2) 

Facilitator between 
districts/schools and external 
support 

1 

3 

8 

Yes (SC-2) 

Provider of direct support to 
districts/ schools 

4 

4 

15 

Yes (SC-2) 

Provider of information about 
federal policies 

9 

7 

29 

No 

Other roles 

1 

1 

3 

No 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

SEA = State Education Agency; SC-2 = Building strong statewide capacity to implement, scale up, and 
sustain the proposed plans. 
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Table B.5. State Provision of Targeted Supports for Implementing Statewide Education Reforms, 
Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported providing targeted 
support to any type of districts or 
schools 

12 

7 

31 

Yes (SC-2) 

Reported providing targeted 
support to the following groups: 
Urban districts or schools 

12 

5 

22 

No 

Rural districts or schools 

11 

5 

25 

No 

Districts or schools with ELLs 

11 

4 

28 

No 

High-minority districts or schools 

11 

5 

20 

No 

High-poverty districts or schools 

12 

7 

27 

No 

Districts or schools that received 
SIG 

12 

7 

31 

No 

Districts or schools “participating” 
in RTT a 

12 

6 

n.a. 

No 

Districts or schools “involved” in 

RTT a 

9 

5 

n.a. 

No 

Districts or schools identified for 
improvement or corrective 
action 

12 

7 

31 

No 

Other types of districts or schools 

5 

3 

14 

No 

Number of States 

10-12 

7 

28-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a These items were asked only in RTT states. 

ELLs = English language learners; SIG = School Improvement Grant; SC-2 = Building strong statewide 
capacity to implement, scale up, and sustain the proposed plans; n.a. = not applicable. 
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Table B.6. State Monitoring of Districts’ Implementation of State Education Reforms, Spring 2012 




Number of States 


Item Aligned 





with RTT 
Application 





Criteria 


Early RTT 

Later RTT 

Non-RTT 

(Subtopic) 

Reported using the following 
strategies to monitor districts’ 
implementation of main state 
education reforms: 





Review of district-submitted 





reports 

11 

6 

31 

No 

Review of third-party reports 
Monitoring site visits by State 

11 

6 

24 

No 

Department of Education staff 
Examination of student 

12 

7 

31 

No 

assessment data by State 
Department of Education staff 

12 

7 

30 

No 

Other strategies 

2 

2 

8 

No 

Reported taking the following 
actions in response to monitoring 
results 





Provided additional resources 
Revised plans for/expectations of 

11 

7 

28 

No 

districts 

11 

7 

31 

No 

Removed state funding/supports 

5 

2 

8 

No 

Provided targeted support 

12 

7 

31 

No 

Other actions 

2 

2 

7 

No 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 
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Table B.7. State Work with Intermediaries to Support Implementation of Statewide Education 
Reforms, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported working with 
intermediaries to support the 
implementation of statewide 
education reforms 

12 

7 

31 

Yes (SC-2) 

Reported working with the following 
types of intermediaries to support 
the implementation of statewide 
education reforms: 

Federally supported 
comprehensive center, 
regional educational 
laboratory, equity assistance 
center, or content center 

11 

6 

31 

No 

Institution of higher education 

11 

6 

30 

No 

Regional/county offices 

8 

5 

14 

No 

Educators contracted by the 
state, such as distinguished 
educators 

9 

6 

25 

No 

Other external organizations 

8 

3 

21 

No 

Other types of intermediary 

1 

1 

6 

No 

Number of States 

11-12 

7 

21-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

SC-2 = Building strong statewide capacity to implement, scale up, and sustain the proposed plans. 


B.9 





Appendix B. Detailed Findings From State Interview’s 


Usage of Policies and Practices Promoted by RTT 


Table B.8. Groups with Which States Expected Intermediaries to Work, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported intermediaries were 
expected to work with the following 
groups: 

State-level staff 

12 

7 

30 

No 

All districts 

7 

4 

18 

No 

Districts identified for 
improvement or corrective 
action under NCLB 

11 

5 

28 

No 

Schools identified for 
improvement, corrective 
action, or restructuring under 
NCLB and/or the districts in 
which these schools are 
located 

12 

5 

27 

No 

Schools that received SIG or 
RTT funds to implement one of 
the four U.S. Department of 
Education-specified school 
intervention models and/or the 
districts in which these schools 
are located 

12 

6 

28 

No 

Districts and/or schools that are 
formally defined as 
“participating” in RTT a 

12 

3 

n.a. 

No 

Districts and/or schools that are 
formally defined as being 
“involved” in RTT a 

7 

3 

n.a. 

No 

Other groups of districts and/or 
schools 

2 

2 

5 

No 

Number of States 

9-12 

6-7 

31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a These items were asked only in RTT states. 

NCLB = No Child Left Behind; SIG = School Improvement Grant; n.a. = not applicable. 
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Table B.9. Structural Changes at the State Education Agency between 2008 and Spring 2012 




Number of States 


Item Aligned 





with RTT 
Application 





Criteria 


Early RTT 

Later RTT 

Non-RTT 

(Subtopic) 

Reported implementing the 
following structural changes at the 
SEA since July 2008 to support 
implementation of state education 
reforms: 





Created new office(s) or 





department(s) 

11 

5 

15 

Yes (SC-2) 

Created new staff positions 
Reorganized the structure of 

12 

7 

25 

Yes (SC-2) 

existing offices/departments 
Changed laws, policies, or 

11 

7 

24 

Yes (SC-2) 

regulations 

Other structural changes to 

11 

7 

26 

Yes (SC-2) 

support the implementation of 
reforms 

3 

2 

10 

No 

Reported the following changes 
since July 2008 to improve the 
SEA’s capacity to address the 
needs of ELLs: 





Increased SEA staff with ELL 
expertise 

Reorganized structure of existing 

2 

2 

11 

No 

offices/departments 

12 

4 

23 

No 

Increased use of consultants with 
ELL expertise 

Increased ELL expertise within 

9 

3 

20 

No 

regional offices 

Redefined policies to better meet 

6 

3 

14 

No 

ELLs’ needs 

7 

3 

22 

No 

Other types of changes 

4 

2 

6 

No 

Number of States 

11-12 

7 

20-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

SEA = State Education Agency; ELLs = English language learners; SC-2 = Building strong statewide 
capacity to implement, scale up, and sustain the proposed plans. 


B.l 1 





Appendix B. Detailed Findings From State Interview’s 


Usage of Policies and Practices Promoted by RTT 


Table B.10. State Coordination of RTT and/or SIG with Other Federal Programs between 2008 and 
Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported coordinating RTT and/or 
SIG with other federal programs in 
the following ways since July 2008: 
Encouraged cofunding of 
initiatives using funds from 
different federal programs 

12 

7 

27 

No 

Established common planning 
activities for state Department 
of Education staff responsible 
for different federal programs 

11 

7 

27 

No 

Increased communication across 
state Department of Education 
staff responsible for different 
federal programs 

12 

7 

28 

No 

Other steps to coordinate RTT 
and/or SIG and other federal 
programs 

5 

4 

14 

No 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

SIG = School Improvement Grant. 
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Table B.11. State-Level Gaps in Expertise to Support Education Reforms, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported gaps in the following 
areas of expertise: 

Monitoring districts and/or 
schools 

1 

1 

6 

No 

Providing targeted support or 
technical assistance to districts 
and/or schools 

2 

2 

11 

No 

Developing and working with 
districts to implement teacher 
evaluation models 

0 

3 

10 

No 

Developing state longitudinal 
data systems 

1 

2 

6 

No 

Working with districts and/or 
schools on the use of data to 
improve instruction 

3 

2 

9 

No 

Supporting districts and/or 
schools in the process of 
turning around low-achieving 
schools 

3 

2 

13 

No 

Other types of expertise 

2 

1 

4 

No 

Number of States 

12 

7 

30-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 
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Table B.12. Barriers Constraining State Educational Agency Capacity to Implement Statewide 
Education Reforms, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported the following factors as 
most constraining SEA's capacity 
to implement statewide education 
reforms: 

Too few SEA staff 

3 

0 

9 

No 

SEA staff working in silos 

2 

0 

1 

No 

Limited expertise in reform topics 

0 

1 

2 

No 

Staff turnover 

1 

0 

2 

No 

Limited state funding 

1 

2 

10 

No 

Limited federal funding 

0 

1 

0 

No 

Need for better technology to 
support reform efforts 

0 

0 

0 

No 

Provisions within collective 
bargaining agreements 

2 

0 

1 

No 

State regulations/laws 

1 

1 

2 

No 

Federal regulations/laws 

0 

0 

1 

No 

Governance structure of state 

0 

0 

0 

No 

Level of support from governor 

0 

0 

0 

No 

Level of state legislature support 

0 

0 

1 

No 

Other factors 

2 

2 

2 

No 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

SEA = State Education Agency. 
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Table B.13. State Adoption of the Common Core State Standards, Spring 2012 



Early RTT 

Number of States 
Later RTT 

Non-RTT 

Item Aligned 
with RTT 
Application 
Criteria 
(Subtopic) 

Reported adopting the CCSS in 
the following subject areas: 

Both ELA and math 

11 

7 

25 

Yes (SA-1) 

Math only 

0 

0 

0 

Yes (SA-1) 

ELA only 

1 

0 

3 

Yes (SA-1) 

Reported supplementing the 
CCSS with state-specific 
standards in the following subject 
areas: 3 

Both ELA and math 

2 

2 

5 

Yes (SA-1) 

Math only 

0 

0 

0 

Yes (SA-1) 

ELA only 

1 

1 

2 

Yes (SA-1) 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 


a States that reported they did not adopt the CCSS in either Math or English are included in the analysis of 
this question as “no” responses. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-1 = Developing and adopting 
common standards. 
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Table B.14. Changes to State Policies and Practices in Response to Adopting the Common Core 
State Standards, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported the following changes to 
state policies and practices in 
response to adopting the CCSS: 
Required use of new curricula or 
textbooks 

0 

0 

4 

Yes (SA-3) 

Required use of new, state- 
approved interim assessments 

1 

0 

0 

Yes (SA-3) 

Changed credit or course 
requirements 

3 

1 

3 

Yes (SA-3) 

Changed the content of the 
state’s high school exit exam 

0 

1 

1 

Yes (SA-3) 

Changed the performance 
standard on the high school 
exit exam 

0 

0 

2 

Yes (SA-3) 

Changed the college entrance 
requirements for the state 
college/university system 

0 

0 

2 

Yes (SA-3) 

Changed policies related to 
teachers and/or school 
leaders, such as licensure, 
certification, or annual 
professional development 
requirements 

5 

0 

6 

Yes (SA-3) 

Other changes 

5 

2 

10 

No 

Number of States 

6-12 

4-7 

15-30 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition 
to enhanced standards and high-quality assessments. 
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Table B.15. State Participation in Consortia to Develop Assessments Aligned to Common Core 
State Standards, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported participating in a 
consortium to develop 
assessments aligned to these 
standards 

12 

7 

27 

Yes (SA-2) 

Reported participating in the 
following consortia: 

Partnership for Assessment of 
Readiness for College and 
Careers (PARCC) only 

9 

5 

5 

No 

SMARTER Balanced 
Assessment Consortium 
(SBAC) only 

3 

0 

19 

No 

Both PARCC and SBAC 

0 

2 

2 

No 

Other consortium 

0 

0 

1 

No 

Number of States 

12 

7 

28 



Source: Interviews with state administrators in spring 2012. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-2 = Developing and 
implementing common, high-quality assessments. 
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Table B.16. Funds Provided by States to Districts and/or Schools to Support Implementation of 
Common Core State Standards, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported providing funds to 
districts and/or schools to support 
implementation of the CCSS since 
adopting them 

8 

1 

8 

Yes (SA-3) 

Reported designating such funds 
for: 

Professional development 

8 

1 

5 

Yes (SA-3) 

Curriculum/textbook purchase 

3 

0 

2 

Yes (SA-3) 

Development or purchase of 
interim assessments or item 
banks 

5 

0 

1 

Yes (SA-3) 

Purchase of hardware or 
software needed to implement 
standards or assessments 

4 

0 

3 

Yes (SA-3) 

Fund additional staff either 
externally contracted or hired 
internally 

6 

0 

2 

Yes (SA-3) 

Other purpose 

4 

0 

2 

No 

Number of States 

12 

7 

27 



Source: Interviews with state administrators in spring 2012. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition 
to enhanced standards and high-quality assessments. 
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Usage of Policies and Practices Promoted by RTT 


Table B.17. State Investments in New Technology to Assist with Implementation of the Common 
Core State Standards, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported investing in new 
technology to assist with 
implementation of the CCSS since 
adopting them 

7 

3 

7 

Yes (SA-3) 

Reported the following technology 
investments for this purpose: 
Hardware 

6 

0 

4 

No 

Software 

6 

2 

4 

No 

Connectivity, such as increased 
bandwidth or Internet speed 
for school systems 

5 

1 

4 

No 

Other investments 

2 

2 

4 

No 

Number of States 

12 

7 

28 



Source: Interviews with state administrators in spring 2012. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition 
to enhanced standards and high-quality assessments. 
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Usage of Policies and Practices Promoted by RTT 


Table B.18. State Investments in New Technology to Assist With Implementation of the 
Assessments Associated With the Common Core State Standards, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported investing in new 
technology to assist with 
implementation of the 
assessments associated with the 
CCSS since adopting them 

6 

2 

4 

Yes (SA-3) 

Reported the following technology 
investments for this purpose: 
Hardware 

5 

0 

2 

No 

Software 

6 

2 

3 

No 

Connectivity, such as increased 
bandwidth or Internet speed 
for school systems 

5 

1 

3 

No 

Other investments 

0 

1 

1 

No 

Number of States 

12 

7 

28 



Source: Interviews with state administrators in spring 2012. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition 
to enhanced standards and high-quality assessments. 
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Usage of Policies and Practices Promoted by RTT 


Table B.19. Materials Provided by States to Help Practitioners Understand the Common Core State 
Standards and/or Change Instruction Based on the Standards, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported providing materials to 
help practitioners understand the 
CCSS and/or change instruction 
based on the standards since 
adoption of these standards 

12 

7 

28 

Yes (SA-3) 

Reported providing the following 
types of materials for this purpose: 
Documents showing alignment 
between the previous state 
standards and the new 
standards 

10 

7 

26 

No 

Documents showing the 
alignment between summative 
assessments and new 
standards, such as blueprints 

4 

4 

11 

No 

Tools or guidance on providing 
instruction aligned with the 
new standards, such as scope 
and sequence, curriculum 
maps, or frameworks 

11 

7 

21 

No 

Examples of instruction 
consistent with new standards 

10 

5 

22 

No 

Sample lesson plans consistent 
with new standards 

6 

3 

18 

No 

Sample performance tasks for 
formative assessment 
purposes, including rubrics or 
scoring guides 

7 

2 

11 

No 

Sample student work 

4 

2 

7 

No 

Banks of interim assessment 
items aligned to standards 

2 

3 

4 

No 

Walk-through or observation 
protocols to aid in monitoring 
alignment of instruction to new 
standards 

7 

2 

5 

No 

Other types of materials 

6 

0 

17 

No 

Number of States 

12 

6-7 

27-28 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition 
to enhanced standards and high-quality assessments. 
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Appendix B. Detailed Findings From State Interview’s 


Usage of Policies and Practices Promoted by RTT 


Table B.20. State Support to Districts and/or Schools for Implementation of Common Core State 
Standards, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported providing professional 
development, training, or technical 
assistance to districts and/or 
schools to support implementation 
of the CCSS since adoption of 
these standards 

12 

7 

27 

Yes (SA-3) 

Reported providing the following 
types of support for this purpose: 
Training on building awareness/ 
understanding of the 
standards 

12 

7 

27 

No 

Training on teaching strategies, 
activities, or methods aligned 
with the standards 

12 

5 

25 

No 

Training on the assessments 
aligned to the standards 

6 

4 

17 

No 

Other types of support 

4 

0 

7 

No 

Number of States 

12 

7 

28 



Source: Interviews with state administrators in spring 2012. 

CCSS = Common Core State Standards; ELA = English/language arts; SA-3 = Supporting the transition 
to enhanced standards and high-quality assessments. 
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Usage of Policies and Practices Promoted by RTT 


Table B.21. State Support to Districts and/or Schools to Aid in Implementation of Common Core 
State Standards with English Language Learners, Spring 2012 



Among the States that Adopted the CCSS in ELA 
and/or Math, Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported providing supports to 
districts and/or schools specifically 
designed to aid in the 
implementation the CCSS with 
ELLs since adoption of these 
standards 

11 

5 

18 

Yes (SA-3) 

Reported providing the following 
support for this purpose: 

Funds 

4 

1 

5 

No 

Materials to support 
understanding and use of new 
standards with ELLs 

7 

4 

15 

No 

Professional development, 
training, or technical 
assistance to support the 
instruction of ELLs 

8 

5 

15 

No 

Mapped new ELA standards to 
state English language 
proficiency standards or 
revised state English language 
proficiency standards for 
better alignment 

6 

4 

15 

No 

Other support 

1 

0 

4 

No 

Number of States 

12 

7 

26-27 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

CCSS = Common Core State Standards; ELA = English/language arts; ELLs = English language 
learners; SA-3 = Supporting the transition to enhanced standards and high-quality assessments. 
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Usage of Policies and Practices Promoted by RTT 


Table B.22. State Implementation of Statewide Longitudinal Data Systems, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported having an SLDS 

12 

7 

28 

Yes (DA-1) 

Reported that the following data 
systems are linked to the state’s K- 
12 education longitudinal data 
system: 

Early childhood 

10 

5 

16 

Yes (DA-1) 

Higher education 

9 

5 

18 

Yes (DA-1) 

Teacher preparation programs 

4 

2 

13 

No 

Health (for example, access to 
medical or mental health 
services) 

3 

2 

5 

No 

Welfare/foster care/child 
protective services 

3 

0 

12 

No 

Workforce (for example, 
employment data) 

3 

0 

9 

No 

Juvenile justice 

3 

3 

10 

No 

Other data systems 

3 

1 

8 

No 

Number of States 

11-12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

SLDS = Statewide Longitudinal Data System; DA-1 = Fully implementing a statewide longitudinal data 
system. 
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Usage of Policies and Practices Promoted by RTT 


Table B.23. State Requirements Related to District Data Systems, Spring 2012 


Number of States 



Early 

RTT 

Later 

RTT 

Non- 

RTT 

Item Aligned with RTT 
Application Criteria 
(Subtopic) 

Reported requiring districts to 
implement district data systems a 

6 

1 

8 

Yes (DA-3) 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

a District data systems (also identified as local instructional improvement systems) are defined in the RTT 
application as technology-based tools or other strategies that provide teachers, principals, and 
administrators with meaningful support and actionable data to systemically manage continuous 
instructional improvement, including such activities as instructional planning; gathering information (for 
example, through formative, interim, and summative assessments, and looking at student work and other 
student data); analyzing information with the support of rapid-time reporting; using this information to 
inform decisions on appropriate next instructional steps; and evaluating the effectiveness of the actions 
taken. Such systems promote collaborative problem solving and action planning; they may also integrate 
instructional data with student-level data, such as attendance, discipline, grades, credit accumulation, and 
student survey results, to provide early-warning indicators of a student’s risk of educational failure. 

DA-3 = Using data to improve instruction. 
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Usage of Policies and Practices Promoted by RTT 


Table B.24. Access to State Longitudinal Data Systems 



Among the States that Reported Having a 
Statewide Longitudinal Data System, Number 
of States 

Item Aligned with 
RTT Application 
Criteria 
(Subtopic) 


Early RTT 

Later RTT 

Non-RTT 

Reported that the following 
groups have access to data from 
the SLDS: 

State Department of Education 
staff in: 

2007-2008 3 

8 

5 

20 

Yes (DA-2) 

2011-2012 

8 

6 

21 


District staff in: 
2007-2008 3 

8 

4 

14 

Yes (DA-2) 

2011-2012 

8 

5 

21 


Principals in: 
2007-2008 3 

7 

3 

11 

Yes (DA-2) 

2011-2012 

8 

5 

20 


Teachers in: 
2007-2008 3 

5 

3 

10 

Yes (DA-2) 

2011-2012 

7 

5 

19 


Colleges and universities in: 
2007-2008 3 

4 

2 

7 

Yes (DA-2) 

2011-2012 

7 

5 

13 


External researchers in: 
2007-2008 3 

6 

2 

8 

Yes (DA-2) 

2011-2012 

8 

6 

15 


Other groups in: 
2007-2008 3 

1 

1 

6 

No 

2011-2012 

5 

2 

13 


Number of States 

8 

5-6 

21 



Source: Interviews with state administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 

SLDS = Statewide Longitudinal Data System; DA-2 = Accessing State data and using it to inform key 
stakeholders. 


B.26 




Appendix B. Detailed Findings From State Interview’s 


Usage of Policies and Practices Promoted by RTT 


Table B.25. Access to the SLDS 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Among the states that reported 
having an SLDS in spring 2012, 
reported tracking usage of their 
SLDS 

8 

4 

15 

No 

Among states that reported having 
an SLDS in spring 2012, reported 
tracking usage of the SLDS by 
user type 

4 

2 

5 

No 

Among the states that reported 
tracking usage of their SLDS by 
user type, mean percentage of 
each type of user who accessed 
the system in the 2010-2011 
school year: 

State Department of Education 
staff 

1 

1 

1 

No 

District staff 

0 

NA 

1 

No 

Principals 

1 

1 

0 

No 

Teachers 

1 

1 

0 

No 

Other groups 

NA 

0 

NA 

No 

Number of States 

2-12 

1-7 

1-28 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

SLDS = Statewide Longitudinal Data System; NA = not available. 
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Usage of Policies and Practices Promoted by RTT 


Table B.26. Uses of Statewide Longitudinal Data by State Staff, Spring 2012 




Number of States 


Item Aligned 





with RTT 
Application 





Criteria 


Early RTT 

Later RTT 

Non-RTT 

(Subtopic) 

Among states that reported having 
an SLDS, reported state-level staff 
use data in the SLDS for the 
following purposes: 





To track overall school 





performance and identify areas 
for improvement 

11 

6 

25 

Yes (DA-2) 

To evaluate instructional programs, 





such as measuring program 
effectiveness 

10 

5 

17 

Yes (DA-2) 

To inform professional 





development offerings, such as 
identifying specific content or 
skills where teachers need 
assistance or support 

6 

1 

11 

Yes (DA-2) 

To evaluate the success of 





professional development 
offerings for teachers or 
principals 

3 

1 

7 

Yes (DA-2) 

To inform resource allocation, such 





as which schools and students 
receive which programs or which 
staff work with which students 

7 

4 

14 

Yes (DA-2) 

To provide information to teachers 
about their students’ progress 

To provide information to parents 

9 

5 

18 

Yes (DA-2) 

about the school or their children 

8 

5 

15 

Yes (DA-2) 

To provide information to students 
about their own progress 

To track students’ postsecondary 

3 

3 

6 

Yes (DA-2) 

enrollment and progress after 
high school graduation, such as 
credits earned in public colleges 
or universities in the state 

7 

5 

15 

Yes (DA-2) 

To provide information to federal 





agencies (for example, EDFacts) 

10 

6 

24 

Yes (DA-2) 

Other purposes 

3 

2 

9 

No 

Number of States 

12 

6-7 

28 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

SLDS = Statewide Longitudinal Data System; DA-2 = Accessing State data and using it to inform key 
stakeholders. 
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Table B.27. State Support for District Data Use, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Provided funding, materials, training, 
technical assistance, or other supports 
to districts to encourage use of data to 
improve instruction 

12 

4 

29 

Yes (DA-3) 

Provided the following supports to 
districts to encourage use of data to 
improve instruction: 

Funds 

10 

3 

12 

No 

Materials or documents on use of 
data to improve instruction 

11 

4 

23 

No 

Technical assistance 

10 

3 

29 

No 

Professional development or training 

10 

3 

24 

No 

Other types of support 

5 

1 

9 

No 

Provided funding, materials, training, 
technical assistance, or other supports 
to districts specifically to aid in use of 
data related to ELLs 

7 

6 

18 

No 

Provided following supports to districts 
to aid in use of data related to ELLs® 
Funds 

2 

2 

8 

No 

Materials or resources such as 
documents or software 

4 

5 

16 

No 

Professional development, training, 
or technical assistance 

7 

6 

18 

No 

Other types of support 

1 

3 

4 

No 

Number of States 

11-12 

7 

30-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

ELLs = English language learners; DA-3 = Using data to improve instruction. 
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Table B.28. Access to Individual Student-Level Data in State Longitudinal Data Systems, Spring 
2012 



Among States that Reported Having a SLDS, 
Number of States 

Item Aligned 
with RTT 


Early RTT 

Later RTT 

Non-RTT 

Application 

Criteria 

(Subtopic) 

Reported that the following groups 
have access to individual student- 
level data for students in the state 
from the SLDS 

State Department of Education 
staff 

12 

7 

25 

No 

District staff 

12 

7 

27 

No 

Principals 

10 

5 

25 

No 

Teachers 

9 

4 

21 

No 

Colleges and universities 

3 

2 

5 

No 

External researchers 

6 

5 

12 

No 

Other groups 

0 

1 

7 

No 

Number of States 

5-12 

2-7 

15-28 



Source: Interviews with state administrators in spring 2012. 

SLDS = Statewide Longitudinal Data System. 
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Table B.29. Authorization of Alternative-Route Certification Programs for Teachers 




Number of States 


Item Aligned with 


Early RTT 

Later RTT 

Non-RTT 

RTT Application 
Criteria 
(Subtopic) 

Reported authorizing AC programs 
for teachers in: 

2007-2008® 

11 

6 

25 

Yes (TL-1) 

2011-2012 

12 

7 

30 


Reported authorizing the following 
groups to operate AC programs for 
teachers: 

Institutions of higher education in: 
2007-2008® 

11 

6 

21 

Yes (TL-1) 

2011-2012 

12 

7 

26 


School districts in: 
2007-2008® 

7 

3 

6 

Yes (TL-1) 

2011-2012 

10 

4 

12 


Educational service districts or 
cooperatives in: 

2007-2008® 

3 

3 

5 

Yes (TL-1) 

2011-2012 

5 

4 

9 


Nonprofit organizations in: 
2007-2008® 

6 

3 

10 

Yes (TL-1) 

2011-2012 

10 

7 

18 


Partnerships or collaborations 
among above groups in: 
2007-2008® 

4 

4 

11 

Yes (TL-1) 

2011-2012 

9 

6 

15 


Other groups or organizations in: 
2007-2008® 

0 

1 

10 

No 

2011-2012 

3 

2 

14 


Number of States 

10-12 

7 

29-31 



Source: Interviews with state administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

Alternative-route certification programs are those that offer an alternative route to 
certification. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
AC = Alternative-route certification; TL-1 = Providing high-quality pathways to certification for aspiring 
teachers and principals. 
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Table B.30. Changes Made to State Regulations Related to Teacher Preparation or Certification 
Programs within the Year Prior to Spring 2012 




Number of States 


Item Aligned with 


Early RTT 

Later RTT 

Non-RTT 

RTT Application 
Criteria 
(Subtopic) 

Reported increasing the selectivity 
of teacher preparation programs 
within the past year 

3 

2 

3 

Yes (TL-1) 

Reported increasing the amount of 
time students in teacher preparation 
programs spend in supervised, 
school-based learning experiences 
within the past year 

3 

1 

3 

Yes (TL-1) 

Reported increasing the amount of 
mentoring and/or coaching that 
students in teacher preparation 
programs receive within the past 
year 

2 

1 

4 

Yes (TL-1) 

Reported allowing AC programs for 
teachers to award the same type of 
certification that traditional 
preparation programs award within 
the past year 

4 

2 

5 

Yes (TL-1) 

Reported other changes within the 
past year 

4 

1 

8 

No 

Number of States 

11-12 

6-7 

28-31 



Source: Interviews with state administrators in spring 2012. 

AC = Alternative-route certification; TL-1 = Providing high-quality pathways to certification for aspiring 
teachers and principals. 
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Table B.31. States’ Assessment of Alternative-Route Teacher Certification Programs 


- 

Early RTT 

Number of States 
Later RTT 

Non-RTT 

Item Aligned with RTT 
Application Criteria 
(Subtopic) 

Reported having a process for 
assessing the effectiveness of AC 
programs for teachers in: 
2007-2008® 

9 

5 

14 

Yes (TL-2) 

2011-2012 

10 

6 

15 


Reported using the following 
information to assess the effectiveness 
of AC programs for teachers: 

The percentage of enrollees who 
earn certification 

5 

3 

9 

No 

The percentage of enrollees placed 
in teaching jobs 

5 

0 

5 

No 

Rates of retention in the profession 

4 

0 

4 

No 

The effectiveness ratings (based in 
part on student growth) of 
credentialed teachers from each 
program 

2 

1 

1 

Yes (TL-2) 

Qualitative program reviews 

8 

5 

13 

No 

Feedback from principals, other 
school staff, or human resources 
staff on credentialed teachers 
from each program 

7 

2 

13 

No 

Other information 

2 

1 

6 

No 

Reported using results from 
assessments of AC programs for 
teachers in the following ways: 

Provided results to certification 
programs 

9 

6 

14 

No 

Provided additional state funding to 
certification programs that were 
shown to be effective 

1 

0 

0 

Yes (TL-2) 

Expanded and/or promoted 
certification programs that were 
shown to be effective 

3 

0 

1 

Yes (TL-2) 

Eliminated and/or reduced state 
funding for certification programs 
that were shown to be ineffective 

2 

0 

0 

No 

Closed programs that were shown 
to be ineffective 

3 

0 

0 

No 

Publically reported results for each 
program 

4 

4 

8 

Yes (TL-2) 

Other actions 

2 

1 

3 

No 

Number of States 

11-12 

7 

28-31 



Source: Interviews with state administrators in spring 2012. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self- 
reported and not independently verified by the research team. For these reasons and potential concerns about 
recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 

AC = Alternative-route certification; TL-2 = Improving teacher and principal effectiveness based on 
performance. 
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Table B.32. Authorization of Alternative-Route Certification Programs for Principals 




Number of States 


Item Aligned with 


Early RTT 

Later RTT 

Non-RTT 

RTT Application 
Criteria 
(Subtopic) 

Reported authorizing AC programs 
for principals in: 

2007-2008® 

6 

4 

9 

Yes (TL-2) 

2011-2012 

10 

6 

17 


Reported authorizing the following 
groups to operate AC programs for 
principals: 

Institutions of higher education in: 
2007-2008® ~ 

6 

4 

8 

Yes (TL-2) 

2011-2012 

10 

6 

13 


School districts in: 
2007-2008® 

5 

1 

3 

Yes (TL-2) 

2011-2012 

7 

2 

5 


Educational service districts or 
cooperatives in: 

2007-2008® 

4 

1 

2 

Yes (TL-2) 

2011-2012 

5 

2 

4 


Nonprofit organizations in: 
2007-2008® 

3 

3 

2 

Yes (TL-2) 

2011-2012 

6 

4 

4 


Partnerships or collaborations 
among above groups in: 
2007-2008® 

5 

2 

3 

Yes (TL-2) 

2011-2012 

9 

3 

8 


Other groups or organizations in: 
2007-2008® 

2 

1 

1 

No 

2011-2012 

3 

2 

4 


Number of States 

10-12 

6-7 

29-30 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
AC = Alternative-route certification; TL-2 = Improving teacher and principal effectiveness based on 
performance. 
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Table B.33. Changes Made to State Regulations Related to Principal Preparation or Certification 
Programs within the Year Prior to Spring 2012 




Number of States 


Item Aligned with 


Early RTT 

Later RTT 

Non-RTT 

RTT Application 
Criteria 
(Subtopic) 

Reported increasing the selectivity 
of principal preparation programs 
within the past year 

4 

1 

2 

Yes (TL-1) 

Reported increasing the amount of 
time students in principal 
preparation programs spend in 
supervised, school-based learning 
experiences within the past year 

2 

1 

4 

Yes (TL-1) 

Reported increasing the amount of 
mentoring and/or coaching that 
students in principal preparation 
programs receive within the past 
year 

2 

2 

5 

Yes (TL-1) 

Reported allowing AC programs for 
principals to award the same type of 
certification that traditional 
preparation programs award within 
the past year 

5 

1 

4 

Yes (TL-1) 

Reported other changes within the 
past year 

1 

2 

7 

No 

Number of States 

12 

6-7 

29-31 



Source: Interviews with state administrators in spring 2012. 

AC = Alternative-route certification; TL-1 = Providing high-quality pathways to certification for aspiring 
teachers and principals. 
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Table B.34. States’ Evaluation of Alternative-Route Principal Certification Programs 




Number of States 


Item Aligned with 


Early RTT 

Later RTT 

Non-RTT 

RTT Application 
Criteria 
(Subtopic) 

Reported having a process for assessing 
the effectiveness of AC programs for 
principals in: 

2007-2008® 

3 

2 

5 

Yes (TL-2) 

2011-2012 

6 

4 

10 


In spring 2012, reported using the 
following information to assess the 
effectiveness of AC programs for 
principals: 

The percentage of enrollees who earn 
certification 

4 

1 

7 

No 

The percentage of enrollees placed in 
school administration jobs 

4 

0 

6 

No 

Rates of retention in the profession 

4 

0 

1 

No 

The effectiveness ratings (based in 
part on student growth) of 
credentialed principals from each 
program 

2 

0 

1 

Yes (TL-2) 

Qualitative program reviews 

5 

4 

9 

No 

Feedback from school, district, or 
human resources staff on 
credentialed principals from each 
program 

5 

1 

8 

No 

Other information 

0 

1 

2 

No 

In spring 2012, reported using results 
from assessments of AC programs for 
principals in the following ways: 
Provided results to certification 
programs 

4 

3 

9 

No 

Provided additional state funding to 
certification programs that were 
shown to be effective 

1 

0 

0 

Yes (TL-2) 

Expanded and/or promoted 
certification programs that were 
shown to be effective 

2 

0 

1 

Yes (TL-2) 

Eliminated and/or reduced state 
funding for certification programs 
that were shown to be ineffective 

0 

0 

0 

No 

Closed programs that were shown to 
be ineffective 

0 

0 

1 

No 

Publically reported results for each 
program 

1 

3 

5 

No 

Other actions 

0 

1 

2 

No 

Number of States 

11-12 

6-7 

30-31 



Source: Interviews with state administrators in spring 2012. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 

AC = Alternative-route certification; TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.35. State Requirements for District Adoption of Teacher Evaluation Models 




Number of States 


Item Aligned 

State-Reported Regulation 

Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Districts must use model prescribed 
by the state 
2007-2008® 

3 

0 

4 

No 

2011-2012 

3 

0 

5 


Districts may adopt state model (state 
exemplar model) 

2007-2008® 

0 

0 

3 

No 

2011-2012 

1 

2 

2 


Districts must adopt state model if 
unable to meet state expectations 
(state default model) 

2007-2008® 

0 

1 

0 

No 

2011-2012 

1 

2 

2 


Districts must administer evaluations 
that comply with state statutes with 
state monitoring evaluation processes 
2007-2008® 

1 

2 

2 

No 

2011-2012 

3 

2 

7 


Districts must administer evaluations 
that comply with state statutes but 
without state monitoring evaluation 
processes 
2007-2008® 

6 

4 

20 

No 

2011-2012 

2 

1 

13 


Number of States 

10 

7 

29 



Source: Interviews with state administrators in spring 2012. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
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Table B.36. Role of Student Achievement Growth in Teacher Evaluations 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported that student achievement 

growth was required 
2007-2008® 

3 

1 

4 

Yes (TL-2) 

2011-2012 

7 

5 

16 


Reported that student achievement 
growth was required with a specific 

weight 

2007-2008® 

1 

0 

0 

Yes (TL-2) 

2011-2012 

7 

4 

10 


Required weight in spring 2012 




No 

No specific weight required or did 
not require student achievement 

growth 

5 

3 

21 


1-20 

1 

0 

1 


21-50 

3 

4 

2 


51 or more 

0 

0 

0 


“Significant,” “substantial,” or 

“primary” factor 

3 

0 

7 


Number of States 

11-12 

7 

31 



Source: Interviews with state administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
TL-2 = Improving teacher and principal effectiveness based on performance. 


B.38 





Appendix B. Detailed Findings From State Interview’s 


Usage of Policies and Practices Promoted by RTT 


Table B.37. State-Reported Requirements for Performance Measures (Other than Student 
Achievement Growth) for Evaluations of Teachers in Tested Grades and Subjects 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Classroom observations 
2007-2008® 

9 

7 

16 

Yes (TL-2) 

2011-2012 

11 

7 

21 


Self-assessment 

2007-2008® 

2 

0 

3 

Yes (TL-2) 

2011-2012 

4 

1 

6 


Portfolios or other artifacts of teacher 
practice 
2007-2008® 

3 

1 

3 

Yes (TL-2) 

2011-2012 

2 

1 

6 


Peer assessments other than 
classroom observations 
2007-2008® 

0 

1 

0 

Yes (TL-2) 

2011-2012 

1 

1 

1 


Student work samples 
2007-2008® 

0 

0 

1 

Yes (TL-2) 

2011-2012 

2 

1 

3 


Student surveys or other feedback 
2007-2008® 

0 

0 

1 

Yes (TL-2) 

2011-2012 

1 

1 

2 


Parent surveys or other feedback 
2007-2008® 

1 

0 

0 

Yes (TL-2) 

2011-2012 

2 

1 

3 


Other measures 
2007-2008® 

0 

1 

2 

No 

2011-2012 

3 

0 

3 


Number of States 

11 

7 

28-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.38. State-Reported Requirements for Performance Measures (Other than Student 
Achievement Growth) for Evaluations of Teachers in Nontested Grades and Subjects 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Classroom observations 
2007-2008 a 

9 

7 

16 

Yes (TL-2) 

2011-2012 

11 

7 

21 


Self-assessment 

2007-2008® 

2 

0 

3 

Yes (TL-2) 

2011-2012 

2 

1 

6 


Portfolios or other artifacts of teacher 
practice 
2007-2008® 

3 

1 

3 

Yes (TL-2) 

2011-2012 

1 

1 

6 


Peer assessments other than 
classroom observations 
2007-2008® 

0 

1 

0 

Yes (TL-2) 

2011-2012 

0 

1 

1 


Student work samples 
2007-2008® 

0 

0 

1 

Yes (TL-2) 

2011-2012 

1 

1 

3 


Student surveys or other feedback 
2007-2008® 

0 

0 

1 

Yes (TL-2) 

2011-2012 

0 

1 

2 


Parent surveys or other feedback 
2007-2008® 

1 

0 

0 

Yes (TL-2) 

2011-2012 

1 

1 

3 


Other measures 
2007-2008® 

0 

1 

2 

No 

2011-2012 

2 

1 

4 


Number of States 

11 

7 

31 



Source: Interviews with state administrators in spring 2012. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.39. Rating Levels for Overall Teacher Performance, Spring 2012 



Number of States (unless otherwise 
specified) 

Item Aligned with 
RTT Application 
Criteria 
(Subtopic) 


Early RTT 

Later RTT 

Non-RTT 

Reported specifying a required 
minimum number of rating levels 

8 

6 

17 

Yes (TL-2) 

Average minimum number of rating 
levels 3 

2.7 

2.6 

1.8 

No 

In spring 2012, reported requiring: 
Four or more levels 

7 

3 

10 

No 

Three levels 

1 

0 

2 


Two levels 

0 

3 

5 


Did not specify a minimum b 

4 

1 

14 


Reported specifying a minimum 
acceptable spread of teachers across 
rating categories 

0 

0 

0 

No 

Number of States 

10-12 

7 

29-31 



Source: Interviews with state administrators in spring 2012. 

a In the analysis of this question, states that reported not specifying a required minimum number of rating 
levels were counted as zeroes. 


b This includes states that responded “no” to the question of “Do state regulations specify a required 
minimum number of rating levels to be used when evaluating overall teacher performance? If so, what is 
the minimum number of rating categories that is required?” 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.40. Tenure and Frequency of Teacher Evaluation 



Number of States (unless otherwise specified) 

Item Aligned 

State-Reported Regulation 

Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Allow teachers to earn tenure 3 
2007-2008 b 

8 

6 

27 

No 

2011-2012 

9 

6 

22 


Have a probationary period for 
teachers 
2007-2008 b 

11 

7 

29 

No 

2011-2012 

11 

7 

26 


Of states that reported having a 
probationary period, mean duration of 
probationary period (years) 
2007-2008 b 

2.6 

3.3 

2.8 

No 

2011-2012 

3.3 

3.3 

3.0 


Evaluated probationary teachers 
At least annually 
2007-2008 b 

10 

6 

22 

Yes (TL-2) 

2011-2012 

10 

6 

24 


Every other year 
2007-2008 b 

0 

0 

0 

No 

2011-2012 

0 

0 

0 


Other interval 
2007-2008 b 

0 

1 

4 

No 

2011-2012 

0 

1 

2 


Evaluated non-probationary teachers 

At least annually 
2007-2008 b 

5 

3 

11 

Yes (TL-2) 

2011-2012 

6 

3 

13 


Every other year 
2007-2008 b 

2 

0 

1 

No 

2011-2012 

2 

1 

2 


Other interval 
2007-2008 b 

3 

4 

15 

No 

2011-2012 

2 

3 

12 


Number of States 

10-12 

7 

24-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a This category includes states that provide the equivalent of tenure but do not use the term “tenure.” 
b Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.41. States’ Use of Teacher Evaluation Results, Spring 2012 




Number of States 


Item Aligned 





with RTT 
Application 





Criteria 

State Regulation 

Early RTT 

Later RTT 

Non-RTT 

(Subtopic) 

Reported requiring results to guide 
decisions about: 





Professional development and 





support 

7 

4 

9 

Yes (TL-2) 

Dismissal 

7 

6 

8 

Yes (TL-2) 

Salary increases 

Bonuses or performance-based 

2 

1 

3 

Yes (TL-2) 

compensation other than salary 

1 

1 

3 

Yes (TL-2) 

Career advancement 

1 

0 

0 

Yes (TL-2) 

Other decisions 

2 

3 

4 

No 

Reported that evaluation results, 
rather than seniority, are the primary 
consideration in reductions in force 
and excessing decisions 

1 

2 

5 

No 

Reported collecting information on the 
percentage of teachers falling in each 
rating level 

8 

4 

12 

No 

Reported analyzing the relationship 
between teacher ratings and student 
achievement growth 

1 

2 

2 

No 

Reported providing teacher 
preparation programs with information 
about effectiveness based on: 




No 

Overall teacher ratings 

1 

0 

1 


Classroom observations 

0 

0 

1 


Student achievement growth 

3 

0 

0 


Number of States 

7-12 

4-7 

21-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.42. State Requirements for District Adoption of Principal Evaluation Models 




Number of States 


Item Aligned 

State-Reported Regulation 

Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Districts must use model prescribed 
by the state 
2007-2008® 

3 

0 

4 

No 

2011-2012 

3 

0 

5 


Districts may adopt state model (state 
exemplar model) 

2007-2008® 

0 

0 

3 

No 

2011-2012 

0 

0 

3 


District must adopt state model If 
unable to meet state expectations 
(state default model) 

2007-2008® 

0 

1 

0 

No 

2011-2012 

0 

2 

1 


Districts must administer evaluations 
that comply with state statutes with 
state monitoring evaluation process 
2007-2008® 

2 

3 

1 

No 

2011-2012 

2 

2 

4 


Districts must administer evaluations 
that comply with state statutes but 
without state monitoring evaluation 
process 
2007-2008® 

4 

1 

19 

No 

2011-2012 

4 

1 

14 


Number of States 

9 

5 

27 



Source: Interviews with state administrators in spring 2012. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
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Table B.43. Role of Student Achievement Growth in Principal Evaluations 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported that student achievement 

growth was required 
2007-2008® 

2 

1 

2 

Yes (TL-2) 

2011-2012 

5 

2 

11 


Reported that student achievement 
growth was required with a specific 

weight 

2007-2008® 

2 

0 

0 

Yes (TL-2) 

2011-2012 

7 

2 

5 


Required weight in spring 2012 




No 

No specific weight required or did 
not require student achievement 

growth 

5 

5 

26 


1-20 

1 

0 

0 


21-50 

0 

0 

0 


51 or more 

0 

0 

0 


“Significant,” “substantial,” or 

“primary” factor 

3 

0 

3 


Number of States 

9-12 

5-7 

30-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.44. State-Reported Requirements for Performance Measures for Principal Evaluations 
(Other than Student Achievement Growth) 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Self-assessment 

2007-2008® 

1 

0 

5 

Yes (TL-2) 

2011-2012 

1 

0 

7 


District administrator input 
2007-2008® 

4 

2 

10 

Yes (TL-2) 

2011-2012 

5 

3 

12 


Staff surveys or other feedback 
2007-2008® 

1 

0 

3 

Yes (TL-2) 

2011-2012 

2 

0 

5 


Student surveys or other feedback 
2007-2008® 

0 

0 

1 

Yes (TL-2) 

2011-2012 

0 

0 

3 


Other measures 
2007-2008® 

3 

1 

4 

No 

2011-2012 

5 

0 

9 


Number of States 

10 

5 

29-30 



Source: Interviews with state administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.45. Rating Levels for Overall Principal Performance 


Reported specifying a required 
minimum number of rating levels 
2007-2008 3 
2011-2012 

Average minimum number of rating 
levels 13 
2007-2008 3 
2011-2012 

In spring 2012, reported requiring: 
Four or more levels 
Three levels 
Two levels 

Did not specify a minimum 0 

Number of States 


Number of States (unless otherwise specified) Item Aligned 

with RTT 

Application 

Criteria 

Early RTT Later RTT Non-RTT (Subtopic 


Yes (TL-2) 

5 2 6 

7 3 14 


1.4 0.6 

2.7 2.3 


7 

0 

1 

4 


3 

0 

2 

2 


0.6 

1.6 

9 

3 

2 

17 


No 


No 


10-12 5-7 31 


Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
b In the analysis of this question, states that reported not specifying a required minimum number of rating 
levels were counted as zeroes. 

c This includes states that responded “no” to the question of “Do state regulations specify a required 
minimum number of rating levels to be used when evaluating overall principal performance? If so, what is 
the minimum number of rating categories that is required?” 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.46. State Uses of Principal Evaluation Results, Spring 2012 




Number of States 


Item Aligned 

State Regulation 

Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported using results to guide 
decisions about: 

Professional development and 
support 

4 

3 

4 

Yes (TL-2) 

Dismissal 

4 

3 

5 

Yes (TL-2) 

Salary increases 

1 

1 

1 

Yes (TL-2) 

Bonuses or performance-based 
compensation other than salary 

1 

0 

1 

Yes (TL-2) 

Career advancement 

1 

0 

0 

Yes (TL-2) 

Other decisions 

1 

1 

1 

No 

Reported collecting information on the 
percentage of principals falling in each 
rating level 

6 

5 

7 

No 

Reported analyzing the relationship 
between principal ratings and student 
achievement growth 

1 

0 

0 

No 

Reported providing principal 
preparation programs with information 
about effectiveness based on: 

Overall principal ratings 

1 

0 

1 

No 

Student achievement growth 

0 

0 

0 

No 

Something else 

0 

0 

1 

No 

Number of States 

10-12 

7 

27-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-2 = Improving teacher and principal effectiveness based on performance. 
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Table B.47. Actions Taken by States to Identify Areas of Teacher Shortages, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported having a process to identify 
areas of teacher shortage 

11 

7 

29 

Yes (TL-3) 

Reported taking actions to identify 
areas of teacher shortage: 

Reviewed the number of vacant 
positions and/or teacher attrition 
by content area, grade, student 
achievement levels, or student 
demographic characteristics 

9 

4 

20 

No 

Reviewed the number of positions 
filled by teachers teaching in 
subjects outside their certification 
area 

11 

4 

25 

No 

Reviewed the projected total 
number of teacher preparation 
program graduates 

8 

0 

16 

No 

Reviewed the projected number of 
teacher preparation program 
graduates in specific subject 
areas 

8 

1 

14 

No 

Reviewed the projected number of 
teacher preparation program 
graduates in specific grade levels 

6 

1 

11 

No 

Surveyed district administrators 

3 

1 

20 

No 

Surveyed school administrators 

1 

1 

8 

No 

Other actions 

5 

2 

9 

No 

Number of States 

12 

7 

30-31 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.48. Full-Time Staff and Actions Taken by States to Address Current Areas of Teacher 
Shortages, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Mean number of full-time staff 3 

152,000 

152,000 

84,000 

No 

Reported having a teacher shortage 

10 

7 

30 

No 

Reported taking actions in 2011-2012 to 
address teacher shortage: 

Enacted policies allowing traditional 
teacher preparation programs to 
expand the number of students 
served 

2 

0 

1 

Yes (TL-3) 

Enacted policies allowing AC programs 
for teachers to expand the number of 
students served 

3 

1 

2 

Yes (TL-3) 

Provided incentives for teachers to 
become certified in areas of shortage 

6 

2 

10 

Yes (TL-3) 

Notified teacher preparation programs 
about the areas of shortage 

9 

5 

20 

Yes (TL-3) 

Encouraged teacher preparation 
programs to train more teachers in 
the areas of shortage 

8 

4 

24 

Yes (TL-3) 

Developed new AC programs for 
teachers focused on the areas of 
shortage 

3 

4 

8 

Yes (TL-3) 

Increased the amount of induction 
support for novice teachers with the 
goal of reducing teacher attrition in 
areas of shortage 

6 

0 

8 

Yes (TL-3) 

Other actions 

1 

0 

10 

No 

Number of States 

9-11 

6-7 

22-31 



Source: Interviews with state administrators in spring 2012, Common Core of Data. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a More specifically, this is the average number of full-time equivalent staff, which means that staff who 
work part time are counted as a proportion of a staff member, rather than as a whole staff member. For 
example, two half-time employees are counted as one full-time equivalent staff member. The mean 
numbers of full-time equivalent staff were rounded to the nearest thousand. 

AC = Alternative-route certification; TL-3 = Ensuring equitable distribution of effective teachers and 
principals. 
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Table B.49. State Analyses of Shifts in Teacher Distribution, Spring 2012 




Number of States 


Item Aligned 





with RTT 
Application 





Criteria 


Early RTT 

Later RTT 

Non-RTT 

(Subtopic) 

Reported conducting analyses to 
determine whether the distribution of 





effective teachers between high- 
poverty or high-minority schools and 
low-poverty or low-minority schools 
shifted 

4 

2 

8 

Yes (TL-3) 

Reported examining factors in the 
analysis of teacher distribution: 





Teacher effectiveness based on 
student achievement growth 
Proportions of novice and 

1 

0 

2 

Yes (TL-3) 

experienced teachers 

2 

2 

6 

No 

Proportion of certified teachers 
Proportion of teachers assigned to 

4 

1 

7 

No 

grades or classes outside of their 
field of certification 

4 

1 

7 

No 

Other factors 

1 

2 

5 

No 

Reported that analyses of distribution 
of effective teachers showed the 





following pattern: 





A more equitable distribution 

1 

0 

3 

No 

A less equitable distribution 

0 

1 

1 

No 

No change in the distribution 

1 

0 

4 

No 

Reported sending results from 
analyses of teacher distribution to the 
districts included in such analyses 3 

4 

2 

7 

No 

Number of States 

10-12 

6-7 

29-30 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.50. State Strategies for Promoting Equitable Distribution of Effective Teachers, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported offering more compensation 
for teachers who work in high-poverty 
or high-minority schools 

3 

1 

3 

Yes (TL-3) 

Reported offering performance-based 
compensation for effective teachers 
who work in high-poverty or high- 
minority schools 

2 

0 

1 

Yes (TL-3) 

Reported hiring staff to work in high- 
poverty or high-minority schools and 
districts earlier 

3 

0 

0 

Yes (TL-3) 

Reported increasing recruitment 
activities 

4 

1 

3 

Yes (TL-3) 

Reported improving teaching and 
learning environments at high-poverty 
or high-minority schools 

2 

1 

4 

Yes (TL-3) 

Reported offering more professional 
development, mentoring, or induction 
for teachers working in high-poverty or 
high-minority schools 

4 

0 

4 

Yes (TL-3) 

Reported other strategies 

1 

2 

6 

No 

Number of States 

12 

7 

30 



Source: Interviews with state administrators in spring 2012. 

TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.51. Actions Taken by States to Identify Areas of Principal Shortages, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported having a process to identify 
areas of principal shortages 

6 

0 

7 

Yes (TL-3) 

Reported taking actions to identify 
areas of principal shortages: 
Reviewed the number of vacant 
positions and/or principal attrition 
by student achievement levels or 
student demographic 
characteristics 

4 

0 

4 

No 

Reviewed the projected total 
number of principal preparation 
program graduates 

5 

0 

6 

No 

Surveyed district administrators 

3 

0 

4 

No 

Surveyed school administrators 

1 

0 

2 

No 

Other actions 

1 

0 

2 

No 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.52. Actions Taken by States to Address Current Areas of Principal Shortages, Spring 2012 


- 


Number of States 


Item Aligned with 
RTT Application 


Early RTT 

Later RTT 

Non-RTT 

Criteria 

(Subtopic) 

Reported having a principal 
shortage 

5 

1 

8 

No 

Reported taking actions in 201 1— 
2012 to address principal shortage: 
Enacted policies allowing 
traditional principal preparation 
programs to expand the number 
of students served 

1 

0 

1 

Yes (TL-3) 

Enacted policies allowing AC 
programs for principals to 
expand the number of students 
served 

2 

0 

2 

Yes (TL-3) 

Provided incentives for individuals 
to become certified to work as 
principals 

2 

0 

1 

Yes (TL-3) 

Notified principal preparation 
programs about the areas of 
shortage 

3 

0 

3 

Yes (TL-3) 

Encouraged principal preparation 
programs to train more 
principals in the areas of 
shortage 

3 

0 

2 

Yes (TL-3) 

Developed new AC programs for 
principals focused on the areas 
of shortage 

4 

1 

2 

Yes (TL-3) 

Increased the amount of support 
for principals with the goal of 
reducing principal attrition in 
areas of shortage 

4 

0 

1 

Yes (TL-3) 

Other actions 

0 

1 

1 

No 

Number of States 

8-10 

6 

25-27 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 

AC = Alternative-route certification; TL-3 = Ensuring equitable distribution of effective teachers and 
principals. 
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Table B.53. State Analyses of Shifts in Principal Distribution, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported conducting analyses to 
determine whether the distribution of 
effective principals between high- 
poverty or high-minority schools and 
low-poverty or low-minority schools 
shifted 

1 

0 

1 

Yes (TL-3) 

Reported examining factors in the 
analysis of principal distribution: 

Principal experience 

1 

0 

1 

Yes (TL-3) 

Principal effectiveness ratings 
based on student achievement 
growth 

1 

0 

0 

No 

Other factors 

0 

0 

1 

No 

Reported that analyses of distribution 
of effective principals showed the 
following pattern: 

A more equitable distribution 

1 

0 

0 

No 

A less equitable distribution 

0 

0 

0 

No 

No change in the distribution 

0 

0 

1 

No 

Reported sending results from 
analyses of principal distribution to the 
districts included in such analyses 3 

1 

0 

1 

No 

Number of States 

12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.54. State Strategies for Promoting Equitable Distribution of Effective Principals, Spring 2012 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported offering more compensation 
for principals who work at high- 
poverty or high-minority schools 

2 

0 

0 

Yes (TL-3) 

Reported offering performance-based 
compensation for effective principals 
of high-poverty or high-minority 
schools 

2 

0 

0 

Yes (TL-3) 

Reported hiring staff in high-poverty or 
high-minority schools and districts 
earlier 

2 

0 

0 

Yes (TL-3) 

Reported increasing external 
recruitment activities 

1 

1 

1 

Yes (TL-3) 

Reported improving teaching and 
learning environments at high-poverty 
or high-minority schools 

2 

1 

3 

Yes (TL-3) 

Reported offering more professional 
development for principals working at 
high-poverty or high-minority schools 

3 

0 

1 

Yes (TL-3) 

Reported other strategies 

1 

0 

2 

No 

Number of States 

11-12 

7 

31 



Source: Interviews with state administrators in spring 2012. 

TL-3 = Ensuring equitable distribution of effective teachers and principals. 
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Table B.55. Organizational Structures Intended to Improve State Capacity to Support School 
Turnaround 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported having organizational or administrative 
structures intended to improve state capacity to 
support school turnaround in: 

2007-2008 3 

7 

6 

12 

Yes (TA-2) 

2011-2012 

12 

7 

26 


Reported having the following organizational or 
administrative structures: 

State office explicitly designated to support school 
turnaround in: 

2007-2008 3 

3 

3 

4 

No 

2011-2012 

12 

5 

11 


Regional offices explicitly designated to support 
school turnaround in: 

2007-2008 3 

1 

0 

1 

No 

2011-2012 

4 

2 

3 


Contracts with external consultants to support 
school turnaround in: 

2007-2008 3 

5 

5 

8 

No 

2011-2012 

8 

5 

16 


State Department of Education staff explicitly 
designated to support school turnaround, but no 
state-level turnaround office in: 

2007-2008 3 

3 

2 

5 

No 

2011-2012 

1 

1 

15 


Regional staff explicitly designated to support 
school turnaround, but no regional state 
turnaround offices, in: 

2007-2008 3 

1 

1 

1 

No 

2011-2012 

3 

3 

6 


State-level staff or consultants to provide support to 
turnaround schools and districts in working with 
ELLs in: 

2007-2008 3 

3 

2 

6 

No 

2011-2012 

6 

4 

13 


Monitoring or reporting requirements specifically for 
schools receiving SIG and/or RTT funds to 
implement a school intervention model in: 
2007-2008 3 

3 

3 

6 

No 

2011-2012 

11 

5 

26 


Other structures reported in: 
2007-2008 3 

1 

2 

1 

No 

2011-2012 

4 

5 

5 


Number of States 

11-12 

5-7 

25-30 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


3 Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported and not 
independently verified by the research team. For these reasons and potential concerns about recall accuracy, readers 
should exercise caution when interpreting data from 2007-2008. 

TA-2 = Turning around the lowest-achieving schools; ELLs = English language learners; SIG = School Improvement Grant. 
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Table B.56. Flexibility Granted by States to Persistently Lowest-Achieving Schools 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported providing their persistently 
lowest-achieving schools with flexibility 
from collective bargaining agreements 
or state policies that guide staffing in: 
2007-2008® 

1 

1 

1 

Yes (TA-2) 

2011-2012 

4 

1 

3 


Reported providing flexibility on the 
following aspects of collective 
bargaining agreements or state 
policies: 

Procedures for assigning or 
removing staff in: 

2007-2008® 

1 

1 

0 

No 

2011-2012 

3 

1 

1 


Requirements or policies related to 
staff hours and responsibilities in: 
2007-2008® 

1 

1 

0 

No 

2011-2012 

3 

0 

2 


Procedures related to the distribution 
of effective staff in: 

2007-2008® 

1 

0 

0 

No 

2011-2012 

2 

0 

1 


Other flexibility in: 
2007-2008® 

1 

1 

1 

No 

2011-2012 

2 

1 

2 


Number of States 

8-9 

7 

26 



Source: Interviews with state administrators in spring 2012. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 

TA-2 = Turning around the lowest-achieving schools. 
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Table B.57. State Teacher Assignment Laws and Policies for Persistently Lowest-Achieving 
Schools 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported having laws, regulations, or 
policies that specifically address 
teacher assignment for their 
persistently lowest-achieving schools 
in: 

2007-2008® 

3 

0 

0 

Yes (TA-2) 

2011-2012 

4 

2 

0 


Reported that their teacher assignment 
laws or policies for their persistently 
lowest-achieving schools have the 
following features: 

Financial and/or other incentives for 
teachers to begin or continue to 
work in the state’s lowest- 
achieving schools in: 

2007-2008® 

2 

0 

0 

Yes (TA-2) 

2011-2012 

4 

2 

0 


Financial and/or other incentives for 
staff with ELL expertise to begin or 
continue to work in the state’s 
lowest-achieving schools in: 
2007-2008® 

0 

0 

0 

No 

2011-2012 

0 

0 

0 


School discretion or authority to 
decide which staff to hire for the 
lowest-achieving schools in: 
2007-2008® 

1 

0 

0 

Yes (TA-2) 

2011-2012 

4 

2 

0 


Teacher tenure rules that affect 
placement in or removal from the 
lowest-achieving schools in: 
2007-2008® 

2 

0 

0 

Yes (TA-2) 

2011-2012 

2 

0 

0 


Other teacher assignment policies 
in: 

2007-2008® 

0 

0 

0 

No 

2011-2012 

0 

0 

0 



Number of States 12 7 30 


Source: Interviews with state administrators in spring 2012. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
TA-2 = Turning around the lowest-achieving schools; ELL = English language learner. 


B.59 




Appendix B. Detailed Findings From State Interview’s 


Usage of Policies and Practices Promoted by RTT 


Table B.58. State Authority to Take Over Failing Schools 




Number of States 


Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

Reported having authority to take over 
failing schools in: 

2007-2008® 

3 

4 

9 

Yes (TA-1) 

2011-2012 

5 

5 

11 


Number of States 

10 

7 

31 



Source: Interviews with state administrators in spring 2012. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
TA-1 = Authority to intervene in the lowest-achieving schools and LEAs. 
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Table B.59. State Restrictions on New Charter Authorization and Charter Enrollment 



Number of States (unless otherwise specified) 

Item Aligned 


Early RTT 

Later RTT 

Non-RTT 

with RTT 
Application 
Criteria 
(Subtopic) 

In spring 2012, number of states that currently permit or 
have ever permitted the authorization of charter schools 

12 

6 

23 

Yes (CH-1) 

Among states that currently permit or have ever permitted 
the authorization of charter schools, percentage of state 
public schools that are charter schools in: 

2007-2008 a 

7.2 

7.0 

3.5 

No 

2011-2012 

9.6 

8.7 

4.5 


Among states that currently permit or have ever permitted 
the authorization of charter schools, percentage of state 
public students that attend charter schools in: 

2007-2008 a 

5.1 

4.3 

2.1 

No 

2011-2012 

7.1 

6.5 

3.4 


Among states that currently permit or have ever permitted 
the authorization of charter schools, number that reported 
having restrictions on the creation of new charter schools or 
charter enrollment in: 

2007-2008 a 

8 

2 

15 

Yes (CH-1) 

2011-2012 

6 

1 

10 


Among states that currently permit or have ever permitted 
the authorization of charter schools, number that reported 
the following restrictions on the creation of new charter 
schools or charter enrollment: 

Caps on total or maximum number of charter schools 
operating overall in: 

2007-2008 3 

5 

2 

7 

No 

2011-2012 

3 

1 

2 


Caps on the number of new charters schools that may be 
authorized in: 

2007-2008 3 

2 

0 

5 

No 

2011-2012 

2 

0 

2 


Restrictions on the number or percentage of students who 
may enroll in charter schools in: 

2007-2008 3 

1 

0 

1 

No 

2011-2012 

1 

1 

2 


Charter schools restricted to specific districts or localities 
in: 

2007-2008 3 

1 

1 

2 

No 

2011-2012 

3 

1 

2 


Moratorium on new charter schools in: 
2007-2008 3 

2 

0 

3 

No 

2011-2012 

1 

0 

0 


Other restrictions reported in: 
2007-2008 3 

3 

1 

2 

No 

2011-2012 

1 

1 

2 


Number of States 

12 

6-7 

20-31 



Source: Interviews with state administrators in spring 2012; National Alliance for Public Charter Schools 

Dashboard [http://dashboard.publiccharters.org/dashboard/home] accessed 8/9/2013. 

Note: A range is provided for the sample sizes because nonresponse varied across items. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were self-reported 
and not independently verified by the research team. For these reasons and potential concerns about recall accuracy, 
readers should exercise caution when interpreting data from 2007-2008. 
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Table B.59 (continued) 

CH-1 = Eliminating restrictions on charter school creation and enrollment. 
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Table B.60. Charter School Authorizers 



Among States That Currently Permit or Have 
Ever Permitted the Authorization of Charter 
Schools, Number of States 

Item Aligned 
with RTT 
- Application 
Criteria 
(Subtopic) 


Early RTT 

Later RTT 

Non-RTT 

Reported the following entities are 
permitted by state law to authorize charter 
schools: 

State department of education and/or 
state board of education in: 

2007-2008® 

7 

5 

13 

No 

2011-2012 

9 

4 

12 


Districts in: 
2007-2008® 

8 

5 

18 

No 

2011-2012 

8 

5 

18 


Regional educational authority in: 
2007-2008® 

2 

0 

2 

No 

2011-2012 

1 

1 

2 


Independent charter school board in: 
2007-2008® 

3 

2 

3 

No 

2011-2012 

2 

2 

5 


Municipal government in: 
2007-2008® 

3 

0 

2 

No 

2011-2012 

1 

1 

3 


Colleges and/or universities in: 
2007-2008® 

4 

1 

6 

No 

2011-2012 

3 

2 

10 


Other non-profit organizations in: 
2007-2008® 

1 

0 

1 

No 

2011-2012 

1 

1 

1 


Number of States 

12 

6 

19-22 



Source: Interviews with state administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
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Table B.61 . State Monitoring of Charter Schools 



Among States That Currently Permit or Have 
Ever Permitted the Authorization of Charter 
Schools, Number of States 

Item Aligned with 
RTT Application 
Criteria 
(Subtopic) 


Early RTT 

Later RTT 

Non-RTT 

Reported having mechanisms in place 
to monitor the performance of charter 
schools in: 

2007-2008® 

11 

6 

20 

Yes (CH-2) 

2011-2012 

12 

6 

20 


Reported monitoring charter school 
performance on the following 
dimensions: 

Academic performance in: 
2007-2008® 

11 

6 

19 

Yes (CH-2) 

2011-2012 

12 

6 

20 


Enrollment in: 
2007-2008® 

9 

4 

15 

Yes (CH-2) 

2011-2012 

10 

4 

15 


Proportion of different student 
populations served in: 
2007-2008® 

8 

4 

9 

Yes (CH-2) 

2011-2012 

10 

4 

9 


Finances and budget in: 
2007-2008® 

11 

6 

18 

Yes (CH-2) 

2011-2012 

12 

6 

19 


Facilities in: 
2007-2008® 

10 

4 

13 

Yes (CH-2) 

2011-2012 

11 

4 

12 


Number of States 

12 

6 

21 



Source: Interviews with state administrators in spring 2012. 


a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 

CH-2 = Refining authorization and monitoring processes for charter schools. 
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Table B.62. Charter School Applications and Authorizations 



Among States That Currently Permit or Have 
Ever Permitted the Authorization of Charter 
Schools, Mean Number (unless otherwise 
specified) 

Item Aligned 
with RTT 
- Application 
Criteria 
(Subtopic) 


Early RTT 

Later RTT 

Non-RTT 

Mean number of charter schools operating 
in the state (2011-2012 school year) 

138 

169 

99 

No 

Mean number of new charter school 
applications submitted in the state in: 
2007-2008® 

22 

24 

8 

No 

2011-2012 

49 

31 

8 


Mean number of new charter schools 
authorized in the state in: 
2007-2008® 

12 

10 

11 

No 

2011-2012 

16 

17 

10 


Mean percentage of yearly charter school 
applications that were authorized in: 
2007-2008® 

34 

48 

58 

No 

2011-2012 

35 

64 

38 


Number of States 

10-12 

4-6 

10-23 



Source: Interviews with state administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
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Table B.63. Charter School Closures 



Among States That Currently Permit or Have 
Ever Permitted the Authorization of Charter 
Schools, Mean Number (unless otherwise 
specified) 

Item Aligned 
with RTT 
■ Application 
Criteria 
(Subtopic) 


Early RTT 

Later RTT 

Non-RTT 

Mean number of charter schools operating 
in the state (2011-2012 school year) 

138 

169 

99 

No 

Mean number of charter schools 
voluntarily closed in the state in: 
2007-2008® 

1.5 

4.8 

0.4 

No 

2011-2012 

1.3 

5.4 

1.3 


Mean number of charter schools whose 
charters were revoked or not renewed in: 
2007-2008® 

0.5 

0.8 

1.2 

No 

2011-2012 

1.1 

2.0 

1.9 


Mean number of charter schools whose 
charters were revoked or not renewed for 
academic reasons in: 

2007-2008® 

0.1 

0.3 

0.1 

No 

2011-2012 

0.7 

2.3 

0.6 


Mean percentage of (involuntary) charter 
school closures that were due to academic 
reasons in: 

2007-2008® 

0 

100 

7 

No 

2011-2012 

75 

100 

7 


Number of States 

2-12 

1-6 

3-23 



Source: Interviews with state administrators in spring 2012. 


Note: A range is provided for the sample sizes because nonresponse varied across items. 

a Data from 2007-2008 were collected retrospectively in spring 2012. All data provided by states were 
self-reported and not independently verified by the research team. For these reasons and potential 
concerns about recall accuracy, readers should exercise caution when interpreting data from 2007-2008. 
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Appendix C. Interview Questions Aligned with RTT Policies and Practices Usage of Policies and Practices Promoted by RTT 


This appendix provides a crosswalk between each policy or practice aligned with the RTT 
application criteria and the state administrator interview questions. For each of the six RTT topic 
areas addressed in Chapter IV, this appendix presents a table showing the interview questions 
that address the area policies and practices aligned with the RTT application criteria. The state 
interview protocol is available at http://www.mathematica-mpr.com/publications/pdfs 
/spring_20 12_state_interview_protocol.pdf. 


Table C.l. Interview Questions Addressing the State Capacity (SC) Topic Area Policies and Practices 


Policy or Practice 

Interview Questions Addressing the State Capacity Policy or Practice 

RTT application criterion 

The state has set forth a comprehensive and coherent reform agenda that clearly articulates its 
goals for implementing reforms in the four education areas described in the ARRA and improving 
student outcomes statewide, establishes a clear and credible path to achieving these goals, and is 
consistent with the specific reform plans that the state proposed throughout its application 

Having a comprehensive 
education reform plan in 
place 

SCI. Does your state have a comprehensive education reform plan in place, which 
means a plan that is intended to result in state-wide improvements in student 
achievement? 

Prioritizing the adoption 
and implementation of 
college and career ready 
standards or assessments 

SC3.To what extent (great extent, moderate extent, little extent, or not at all) is each 
of the following educational reform initiatives a priority in your state for the current 
school year? a. Adopting and implementing college and career ready standards; b. 
Adopting and implementing college and career ready assessments 

Prioritizing the 
development of 
comprehensive, student- 
level, longitudinal data 
systems or using data to 
improve instruction 

SC3.To what extent (great extent, moderate extent, little extent, or not at all) is each 
of the following educational reform initiatives a priority in your state for the current 
school year? c. Building comprehensive, student-level, longitudinal data systems; d. 
Using data to improve instruction 

Prioritizing the recruiting, 
developing, rewarding, 
and retaining of effective 
teachers and principals, 
especially where they are 
needed most (including 
developing and 
implementing evaluation 
systems) 

SC3.To what extent (great extent, moderate extent, little extent, or not at all) is each 
of the following educational reform initiatives a priority in your state for the current 
school year? e. Recruiting effective teachers and school leaders; f. Retaining 
effective teachers and school leaders; g. Rewarding effective teachers and school 
leaders, for example, by offering bonuses and/or increased compensation to effective 
teachers; h. Developing and preparing effective teachers and school leaders; i. 
Developing and implementing a teacher and principal evaluation system that is based 
on student growth; j. Improving the distribution of effective teachers and principals 

Prioritizing the turnaround 
of its lowest-achieving 
schools 

SC3.To what extent (great extent, moderate extent, little extent, or not at all) is each 
of the following educational reform initiatives a priority in your state for the current 
school year? k. Turning around the lowest-achieving schools 

RTT application criteria 

Ensure state has the capacity required to implement its proposed plans by providing strong 
leadership and dedicated teams to implement the statewide education reform plans; supporting 
participating LEAs in successfully implementing the education reform plans the state has 
proposed 

Use support from a broad group of stakeholders to better implement its plans, as evidenced by 
the strength of the statements or actions of support from the state’s teachers and principals and 
other critical stakeholders 

Providing leadership or 
teams to support the 
implementation of reform 
plans 

SC6. To what extent does the state education agency play each of the following roles 
(a great extent, moderate extent, little extent, or not at all)? a. Creator of a state-wide 
vision for reforms such as articulating a vision and direction for educational 
improvement in the state; d. Facilitator between educational units and external 
expertise; e. Supporter providing direct support services to districts and schools? 
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Table C.1 (continued) 


Policy or Practice 

Interview Questions Addressing the State Capacity Policy or Practice 


SC7. Within the past year, has the state education agency provided targeted support 
to certain types of districts or schools for implementation of statewide education 
reform priorities? By targeted supports, we mean supports that are provided to a 
specific group of schools and/or districts for a particular purpose. For example, this 
might include, but is not limited to, the state providing technical assistance to districts 
implementing a new teacher evaluation system. 


SCI 3. Does your state education agency currently work with any intermediaries to 
support the implementation of statewide education reform priorities in any of the 
following areas? For each reform area selected, please briefly describe the role 
assigned to intermediaries in that area. a. Implementing college and career ready 
standards and assessments; b. Using data to improve instruction; c. Recruiting, 
developing, rewarding, and retaining effective teachers and school leaders; d. 
Turning around your state's lowest achieving schools; e. Providing supports for 
English language learners; f. Increasing state capacity in any of the areas just 
mentioned. 

Working with districts that 
have the highest 
achievement levels and 
seeking to replicate their 
practices statewide to 
implement the education 
reform plan 

SC2. Which of the following strategies is your state using to implement this reform 
plan? b. Working with districts that have the highest achievement levels and seeking 
to replicate their practices statewide 

Providing effective and 
efficient operations and 
processes for grant 
oversight and 
performance measure 
tracking and reporting 

SC6. To what extent does the state education agency play each of the following roles 
(a great extent, moderate extent, little extent, or not at all)? b. Creator and monitor of 
performance measures including assessing the extent to which districts and/or 
schools are attaining specific performance goals; c. Compliance monitor of reform 
priorities such as ensuring compliance with state or federal laws and regulations 

SCI 8. Since July 2008, has the state education agency implemented and sustained 
any of the following changes to support the implementation of state education reform 
priorities? These could include reform priorities funded by Race to the Top (if 
applicable), as well as statewide reform efforts funded in other ways. a. Created a 
new office(s) or department(s); b. Created new staff positions; c. Reorganized the 
structure of existing offices/departments; d. Changed laws, policies, or regulations (IF 
YES, SPECIFY) 

Involving teachers, 
teachers unions or 
associations, or school 
administrators in defining 
its education reform 
initiatives or priorities 

SC5. To what extent (great extent, moderate extent, little extent, or not at all) are 
each of the following individuals or groups involved in defining your state’s education 
reform initiatives or priorities? c. Teachers; d. State teachers union or association; i. 
School administrators 

Involving other 
stakeholders in defining 
its education reform 
initiatives or priorities 

SC5. To what extent (great extent, moderate extent, little extent, or not at all) are 
each of the following individuals or groups involved in defining your state’s education 
reform initiatives or priorities? a. Governor’s office; b. State legislature; e. 

Businesses; f. Civil rights leaders; g. Local school boards; h. State school board; j. 
Parent, student and community organizations such as parent teacher organizations, 
local education foundations, community based organizations, or advocacy groups; k. 
Institutes of higher education including teacher and principal preparation programs or 
schools of education 


Source: Interviews with state administrators in spring 2012. 
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Table C.2. Interview Questions Addressing the Standards and Assessments (SA) Topic Area Policies and 
Practices 


Policy or Practice 

Interview Questions Addressing the Standards and Assessments Policy or Practice 

RTT application criterion 

Commitment to adopting a common set of high-quality standards, evidenced by participation in a 
consortium of states that is working jointly to develop and adopt a common set of K-12 
standards 

Adopting the Common 
Core State Standards 
(CCSS) in both 
English/language arts 
and math 

SA1. Did your state adopt the Common Core State Standards in both English/language 
arts and math, math only, English/language arts only, or neither subject? 

SA3. What proportion of your total English/language arts and math standards are state- 
specific (meaning they are not part of the Common Core State Standards)? Please 
report separate percentages for English/language arts and math. 

RTT application criterion 

Commitment to improving the quality of assessments, evidenced by participation in a consortium 
of states that is working jointly to develop and implement common, high-quality assessments 

Participating in a 
consortium of states to 
develop assessments 
aligned to CCSS 

SA7. Is your state participating in a consortium to develop assessments aligned to [the 
Common Core State Standards/your current state standards]? 

Has specified the 
school year by which 
state plans to fully 
implement summative 
assessments being 
developed by the 
consortium 

SA9. In what school year does your state plan to fully implement, as opposed to pilot 
test, the summative assessments in English/language arts and math that are being 
developed by the consortium(a) in which you are participating? Please report the year 
by which you plan to fully implement the summative assessments being developed by 
the consortium in both subjects. 

RTT application criterion 

Supporting the statewide transition to and implementation of K-12 standards that build toward 
college and career readiness by the time of high school graduation and high-quality assessments 
tied to these standards 

Supporting the 
implementation of 
CCSS by providing 
funds for additional 
staff or making new 
technology 
investments 

SA14. For which of the following purposes were these funds designated? e. Fund 
additional staff either externally contracted or hired internally 3 

SA15. Since the adoption of [the Common Core State Standards/your current state 
standards], has the state made investments in new technology to assist with 
implementation of the standards? 

Supporting the 
implementation of the 
assessments 
associated with the 
CCSS by making new 
technology 
investments 

SA17. Since the adoption of [the Common Core State Standards/your current state 
standards], has the state made investments in new technology to assist with 
implementation of the assessments associated with those standards? 

Requiring or 
supporting the use of 
new instructional 
materials for 
implementing the 
CCSS 

SA12. Since the adoption of [the Common Core State Standards/your current state 
standards], has your state made any of the following types of changes to state policies 
and practices in response to the adoption of such standards? a. Required use of new 
curricula or textbooks 

SA14. For which of the following purposes were these funds designated? b. 
Curriculum/textbook purchase 3 


SA19. Since adoption of [the Common Core State Standards/your current state 
standards], has the state provided any materials to help practitioners understand the 
standards and/or change instruction based on the standards? 
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Table C.2 (continued) 


Policy or Practice Interview Questions Addressing the Standards and Assessments Policy or Practice 


Developing, 
supporting, or 
requiring new interim 
assessments 
associated with the 
CCSS 


Changing high school 
exit requirements or 
college entrance 
requirements 


Changing policies 
about or providing 
funds for professional 
development, training, 
and technical 
assistance for 
teachers or school 
leaders to support 
implementation of 
CCSS 


Supporting districts or 
schools in 
implementing the 
CCSS with English 
language learners 


SA12. Since the adoption of [the Common Core State Standards/your current state 
standards], has your state made any of the following types of changes to state policies 
and practices in response to the adoption of such standards? b. Required use of new, 
state-approved interim assessments 

SAM. For which of the following purposes were these funds designated? c. 
Development or purchase of interim assessments or item banks; d. Purchase hardware 

or software needed to implement standards or assessments 3 

SA12. Since the adoption of [the Common Core State Standards/your current state 
standards], has your state made any of the following types of changes to state policies 
and practices in response to the adoption of such standards? c. Changed credit or 
course requirements such as adopting use of competency-based credit for courses or 
subject knowledge; d. Changed the content of the state’s high school exit exam; e. 
Changed the performance standard on the high school exit exam; f. Changed the 

college entrance requirements for the state college/university system 

SA12. Since the adoption of [the Common Core State Standards/your current state 
standards], has your state made any of the following types of changes to state policies 
and practices in response to the adoption of such standards? g. Changed policies 
related to teachers and/or school leaders such as licensure, certification, or annual 
professional development requirements 

SAM. For which of the following purposes were these funds designated? a. 
Professional development 3 

SA21 . Since adoption of [the Common Core State Standards/your current state 
standards], has the state provided any professional development, training, or technical 
assistance (either directly or through an intermediary) to districts and/or schools to 

support implementation of the standards? 

SA25. Since the adoption of [the Common Core State Standards/your current state 
standards], has the state provided any supports to districts and/or schools specifically 
designed to aid in the implementation of the state’s standards with English language 
learners? This might include, but is not limited to, funds, professional development, 
technical assistance, or materials to support use of standards with English language 
learners. 


Source: Interviews with state administrators in spring 2012. 

3 This item followed a screener question that asked “Since the adoption of [the Common Core State Standards/your 
current state standards], has the state provided any funds to districts and/or schools to support implementation of the 
new standards?” 
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Table C.3. Interview Questions Addressing the Data Systems (DA) Topic Area Policies and Practices 

Policy or Practice Interview Addressing Comprising the Data Systems Policy or Practice 

RTT application criterion: 

Fully implement a state longitudinal data system (SLDS) that includes all of the America COMPETES Act 
elements 

Having an SLDS DAI . Does your state currently have a statewide longitudinal data system? 

DAI 6. Which of the following types of data specifically related to English language learner 
students are currently included in your state’s longitudinal data system(s)? a. Students 
currently identified as English language learners; b. Students formerly identified as English 
language learners; c. Students’ English language proficiency test scores; d. English language 
learners’ proficiency in their native (home) language; e. Student participation in bilingual 
education programs, defined as programs that focus on developing literacy in two languages, 
which include non-English speakers and may include English speakers, and which involve 
instruction in English and another language; f. Student participation in other educational 
programs specifically designed for English language learners; g. Student’s native language; 
h. Number of years the student has lived in U.S.; k. The number of years that students have 
received English language instructional services; I. The length of time it took for an English 
language learner to exit English language learners status 

DA2. We are now going to ask you about data systems other than your state’s K-12 system, 
and whether they are linked to your state's K-12 education longitudinal data system, which 
means that data stored in these other systems can be accessed through the K-12 system. 

For each of the following data systems please indicate if they are currently linked to the 
state’s K-12 longitudinal education data system, a. Early childhood 

DA2. We are now going to ask you about data systems other than your state’s K-12 system, 
and whether they are linked to your state's K-12 education longitudinal data system, which 
means that data stored in these other systems can be accessed through the K-12 system. 

For each of the following data systems please indicate if they are currently linked to the 
state’s K-12 longitudinal education data system, b. Higher education 
RTT application criterion: 

Ensure that key stakeholders (for example, principals, teachers, parents, community members, 
administrators) have access to and use data from the SLDS to support decision makers in 
continuous improvement efforts in areas such as policy, instruction, operations, management, 
resource allocation, and overall effectiveness 

SLDS can be DA3. Which of the following groups currently has access to data from the state longitudinal 

accessed by data system? a. State Department of Education; b. district staff; c. principals; d. teachers; e. 

stakeholders colleges and universities; f. external researchers 

DAI 0. For which of the following purposes are data in the state longitudinal data system 
currently used by state-level staff? a. To track overall school performance; b. to evaluate 
instructional programs; c. To inform professional development offerings such as identifying 
specific content or skills where teachers need assistance or support; d. To evaluate the 
success of professional development offerings for teachers or principals; e. To inform 
resource allocation, such as which schools and students receive which programs or which 
staff work with which students; f. To provide information to teachers about their students’ 
progress; g. To provide information to parents about the school or their children; h. To provide 
information to students about their own progress; i. To track students’ postsecondary 
enrollment and progress after high school graduation, such as credits earned in public 
colleges or universities in the state; j. To provide information to federal agencies (for example, 
EDFacts) 

RTT application criteria: 

- Increase the acquisition, adoption, and use of local instructional improvement systems; support 
participating LEAs and schools that use instructional improvement systems in providing effective 
professional development to teachers, principals, and administrators on how to use these systems 

- Make data from instructional improvement systems, together with SLDS data, available to researchers 
to evaluate the effectiveness of instructional materials, strategies, and approaches for educating 
different types of students 

Requiring districts to DA1 1 . Does the state require districts to implement district state systems as defined above? 
implement district (we will refer to local instructional improvement systems as district data systems, which are 
data systems technologically based tools or strategies that provide educators with data to manage 

continuous instructional improvement efforts) 


SLDS is used to 
inform and engage 
stakeholders and 
support decision 
makers in 
continuous 
improvement efforts 


SLDS is linked to a 
higher education 
data system 


SLDS is linked to an 
early childhood data 
system 


SLDS has student- 
level enrollment and 
program 
participation 
information 
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Table C.3 (continued) 


Policy or Practice 

Interview Addressing Comprising the Data Systems Policy or Practice 

Providing funding, 
materials, training, 
technical assistance, 
or other supports to 
districts to 
encourage the use 
of data to improve 
instruction 

DAI 2. Within the past year, has the state provided funding, materials, training, technical 
assistance, or other supports to districts to encourage the use of data to improve instruction? 


Source: Interviews with state administrators in spring 2012. 
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Table C.4. Interview Questions Addressing the Teacher and Principal Certification and Evaluation (TL) Topic 
Area Policies and Practices 


Policy or Practice 

Interview Questions Addressing the Teacher and Principal Certification 
and Evaluation Policy or Practice 

RTT application criteria: 

Legal, statutory, or regulatory provisions that allow alternative routes to certification for teachers and 
principals, particularly routes that allow for providers in addition to higher education institutions 
Alternative routes to teacher and principal certification that are in use 

A process for monitoring, evaluating, and identifying areas of teacher and principal shortage and for 
preparing teachers and principals to fill these areas of shortage 

Authorizing or 
expanding institutions 
qualified to operate 
alternative-route 
programs for teachers 

TL2. Currently, which of the following groups are authorized to operate alternative teacher 
certification programs in your state? a. Institutions of higher education; b. School districts; 
c. Educational service districts or cooperatives in your state; d. Nonprofit organizations 
such as Teach for America; e. Partnerships or collaborations between groups listed above 

TL5. Within the past year, did the state implement any of the following types of changes to 
state regulations or policies related to either traditional teacher preparation programs or 
alternative teacher certification programs in the state? a. Expanded the types of 
institutions qualified to operate teacher preparation programs to include providers 
operating independently of institutions of higher education 

Adopting policies to 
increase the selectivity 
of alternative-route 
programs for teachers 

TL5. Within the past year, did the state implement any of the following types of changes to 
state regulations or policies related to either traditional teacher preparation programs or 
alternative teacher certification programs in the state? b. Adopted policies or regulations 
designed to increase the selectivity of teacher preparation programs 

Adopting policies to 
increase the amount of 
mentoring for 
participants in teacher 
certification programs 
or the time such 
participants spend in 
school-based learning 
experiences 

TL5. Within the past year, did the state implement any of the following types of changes to 
state regulations or policies related to either traditional teacher preparation programs or 
alternative teacher certification programs in the state? c. Adopted policies or regulations 
designed to increase the amount of time students in teacher preparation programs spend 
in supervised, school-based learning experiences; d. Adopted policies or regulations to 
increase the amount of mentoring and/or coaching that students in teacher preparation 
programs receive 

Adopting policies 
allowing alternative- 
route programs for 
teachers to award the 
same type of 
certification as 
traditional preparation 
programs 

TL5. Within the past year, did the state implement any of the following types of changes to 
state regulations or policies related to either traditional teacher preparation programs or 
alternative teacher certification programs in the state? e. Adopted policies or regulations 
that allow alternative certification programs to award the same type of certification that 
traditional preparation programs award 

Groups are currently 
operating alternative- 
route programs for 
teachers 

TL1. Currently, does the state authorize alternative routes to certification for teachers? 

TL2. Currently, which of the following groups are authorized to operate alternative teacher 
certification programs in your state? 


TL3. Are they currently operating such programs in your state? a. Institutions of higher 
education; b. School districts; c. Educational service districts or cooperatives in your state; 
d. Nonprofit organizations such as Teach for America; e. Partnerships or collaborations 
between groups listed above 

Having a process to 
identify areas of 
teacher shortages 

TL14. Does the state currently have a process to identify areas of teacher shortages? 
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Table C.4 (continued) 


Policy or Practice 

Interview Questions Addressing the Teacher and Principal Certification 
and Evaluation Policy or Practice 

Taking steps to 
address areas of 
teacher shortage 

TL17. Within the past year, has the state taken any of the following steps to address these 
areas of teacher shortage? a. Enacted policies allowing traditional teacher preparation 
programs to expand the number of students served; b. Enacted policies allowing 
alternative teacher preparation programs to expand the number of students served; c. 
Provided incentives for teachers to become certified in areas of shortage (SPECIFY); d. 
Notified teacher preparation programs about the areas of shortage; e. Encouraged 
teacher preparation programs to train more teachers in the areas of shortage; f. 

Developed or began developing new alternative teacher certification programs focused on 
the areas of shortage; g. Increased the amount of induction support for novice teachers 
with the goal of reducing teacher attrition in areas of shortage 

Authorizing or 
expanding institutions 
qualified to operate 
alternative-route 
programs for principals 

TL19. Currently, which of the following groups are authorized to operate alternative 
principal certification programs in your state? a. Institutions of higher education; b. School 
districts; c. Educational service districts or cooperatives in your state; d. Nonprofit 
organizations such as Teach for America; e. Partnerships or collaborations between 
groups listed above 


TL22. Within the past year, did the state implement any of the following types of changes 
to state regulations or policies related to either traditional principal preparation programs 
or alternative principal certification programs in the state? a. Expanded the types of 
institutions qualified to operate principal preparation programs to include providers 
operating independently of institutions of higher education 

Adopting policies to 
increase the selectivity 
of alternative-route 
programs for principals 

TL22. Within the past year, did the state implement any of the following types of changes 
to state regulations or policies related to either traditional principal preparation programs 
or alternative principal certification programs in the state? b. Adopted policies or 
regulations designed to increase the selectivity of principal preparation programs 

Adopting policies to 
increase the amount of 
mentoring for 
participants in principal 
certification programs 
or the time such 
participants spend in 
school-based learning 
experiences 

TL22. Within the past year, did the state implement any of the following types of changes 
to state regulations or policies related to either traditional principal preparation programs 
or alternative principal certification programs in the state? c. Adopted policies or 
regulations designed to increase the amount of time students in principal preparation 
programs spend in supervised, school-based learning experiences 

Adopting policies 
allowing alternative- 
route programs for 
principals to award the 
same type of 
certification as 
traditional preparation 
programs 

TL22. Within the past year, did the state implement any of the following types of changes 
to state regulations or policies related to either traditional principal preparation programs 
or alternative principal certification programs in the state? e. Adopted policies or 
regulations that allow alternative certification programs to award the same type of 
certification that traditional preparation programs award 

Groups are currently 
operating alternative- 
route programs for 
principals 

TL18. Currently, does the state authorize or permit alternative routes to certification for 
principals? 

TL19. Currently, which of the following groups are authorized to operate alternative 
principal certification programs in your state? 


TL20. Are they currently operating such programs in your state? a. Institutions of higher 
education; b. School districts; c. Educational service districts or cooperatives in your state; 
d. Nonprofit organizations such as Teach for America; e. Partnerships or collaborations 
between groups listed above 

Having a process to 
identify areas of 
principal shortages 

TL31. Does the state currently have a process to identify areas of principal shortages? 


C.9 




Appendix C. Interview Questions Aligned with RTT Policies and Practices Usage of Policies and Practices Promoted by RTT 


Table C.4 (continued) 


Policy or Practice 

Interview Questions Addressing the Teacher and Principal Certification 
and Evaluation Policy or Practice 

Taking steps to 
address areas of 
principal shortage 

TL34. Within the past year, has the state taken any of the following steps to address these 
areas of principal shortage? a. Enacted policies allowing traditional principal preparation 
programs to expand the number of students served; b. Enacted policies allowing 
alternative principal preparation programs to expand the number of students served; c. 
Provided incentives for principals to become certified in areas of shortage (SPECIFY); d. 
Notified principal preparation programs about the areas of shortage; e. Encouraged 
principal preparation programs to train more principals in the areas of shortage; f. 
Developed or began developing new alternative principal certification programs focused 
on the areas of shortage; g. Increased the amount of induction support for novice 
principals with the goal of reducing principal attrition in areas of shortage 

RTT application criteria: 

Design and implement rigorous, transparent, and fair evaluation systems for teachers and principals 
that differentiate effectiveness using multiple rating categories that take into account data on student 
growth as a significant factor 

Conduct annual evaluations of teachers and principals that include timely and constructive feedback 
Use evaluations, at minimum, to inform decisions regarding: developing teachers and principals, 
including by providing relevant coaching, induction support, or professional development; 
compensating, promoting, and retaining teachers and principals, including by providing 
opportunities for highly effective teachers and principals to obtain additional compensation and be 
given additional responsibilities; removing ineffective tenured and untenured teachers and principals 
after they have had ample opportunities to improve and ensuring that such decisions are made using 
rigorous standards and streamlined, transparent, and fair procedures 

Requiring districts to 
use student growth to 
evaluate teachers and 
specifying the extent to 
which student 
achievement growth 
must factor into 
teacher evaluations 

TL38. Currently, do state regulations specify the extent to which evidence of student 
growth must factor into the overall teacher evaluation? For example, regulations may call 
for it to be a “significant” factor in evaluations or call for student growth to have a specific 
weight such as 20 percent. If this is specified in state regulations, please describe those 
specific regulations. 

[Note: TL36 (shown in the next row) was also used to address the policy in this row. 
Specifically, this policy was coded as 0 if either the response to TL38 was “no” or the 
response to TL36 was “no teachers. ”1 

Requiring multiple 
performance 
measures for teacher 
evaluations 

TL36. Currently, does the state require that districts use student growth as one measure 
when evaluating all teachers, some teachers, or no teachers? 

TL42. Other than the student growth performance measure we just discussed, which of 
the following other measures of teacher performance are currently required by the state 
for teacher evaluations for teachers in tested grades and/or subjects? a. Classroom 
observations conducted by the principal; b. Classroom observations conducted by a 
school administrator other than the principal; c. Classroom observations conducted by 
someone other than a school administrator, such as a peer or mentor teacher; d. self- 
assessment; e. portfolios or other artifacts of teacher practice; f. Peer assessments other 
than classroom observations; g. Student work samples; h. Student surveys or other 
feedback; i. Parent surveys or other feedback 


TL43. Other than the student growth performance measure we just discussed, which of 
the following other measures of teacher performance are currently required by the state 
for teacher evaluations for teachers in non-tested grades and/or subjects? a. Classroom 
observations conducted by the principal; b. Classroom observations conducted by a 
school administrator other than the principal; c. Classroom observations conducted by 
someone other than a school administrator, such as a peer or mentor teacher; d. self- 
assessment; e. portfolios or other artifacts of teacher practice; f. Peer assessments other 
than classroom observations; g. Student work samples; h. Student surveys or other 
feedback; i. Parent surveys or other feedback 

Specifying a required 
minimum number of 
rating categories to be 
used when evaluating 
teachers 

TL44. Do state regulations specify a required minimum number of rating levels to be used 
when evaluating overall teacher performance? If so, what is the minimum number of rating 
categories that is required? 
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Table C.4 (continued) 


Policy or Practice 

Interview Questions Addressing the Teacher and Principal Certification 
and Evaluation Policy or Practice 

Conducting annual 
evaluations of 
teachers 

TL40. Currently, how often does the state require that teachers be evaluated during their 
probationary period? For example, every six months, annually, every other year, or at 
some other interval? 


TL41. Currently, how often does the state require that teachers be evaluated in their non- 
probationary period? For example, every six months, annually, every other year, or at 
some other interval? 

Requiring teacher 
evaluations to inform 
decisions about 
professional 
development and 
support for individual 
teachers 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation results to be 
used for any of the following purposes? Please select just one answer for each potential 
use. If your state both permits and requires that evaluation results be used for a particular 
purpose, please report that purpose as required, a. To guide decisions about professional 
development and support for individual teachers 

Requiring teacher 
evaluations to inform 
decisions about 
compensation 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation results to be 
used for any of the following purposes? Please select just one answer for each potential 
use. If your state both permits and requires that evaluation results be used for a particular 
purpose, please report that purpose as required, c. To guide decisions about annual 
teacher salary increases; d. To guide decisions about teacher bonuses or performance- 
based compensation other than salary 

Requiring teacher 
evaluations to inform 
decisions about career 
advancement 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation results to be 
used for any of the following purposes? Please select just one answer for each potential 
use. If your state both permits and requires that evaluation results be used fora particular 
purpose, please report that purpose as required, e. To guide decisions about career 
advancement opportunities 

Requiring teacher 
evaluations to inform 
decisions about 
dismissal of teachers 

TL60. Currently, does the state prohibit, permit, or require teacher evaluation results to be 
used for any of the following purposes? Please select just one answer for each potential 
use. If your state both permits and requires that evaluation results be used for a particular 
purpose, please report that purpose as required, b. To guide decisions about dismissal of 
teachers 

Requiring districts to 
use student growth to 
evaluate principals and 
specifying the extent to 
which student 
achievement growth 
must factor into 
principal evaluations 

TL75. Currently, do state regulations specify the extent to which evidence of student 
growth must factor into the overall principal evaluation? For example, regulations may call 
for it to be a “significant” factor in evaluations or call for student growth to have a specific 
weight such as 20 percent. If this is specified in state regulations, please describe those 
specific regulations. 

[Note: TL74 (shown in the next row) was also used to address the policy in this row. 
Specifically, this policy was coded as 0 if either the response to TL75 was “no” or the 
response to TL74 was “no principals.”] 

Requiring multiple 
performance 
measures for principal 
evaluations 

TL74. Currently, does the state require that all districts use student growth as one 
measure when evaluating all principals, some principals, or no principals? 

TL77. Other than the student growth measure we just discussed, which of the following 
other measures of principal performance are currently required by the state? a. Self- 
assessment; b. District administrator input; c. Staff surveys or other feedback; d. Student 
surveys or other feedback 

Specifying a required 
minimum number of 
rating categories to be 
used when evaluating 
principals 

TL78. Do state regulations specify a required minimum number of rating levels to be used 
when evaluating principal performance? If so, what is the minimum number of rating 
categories that is required? 

Requiring principal 
evaluations to inform 
decisions about 
professional 
development and 
support for individual 
principals 

TL89. Currently, does the state prohibit, permit, or require principal evaluation results to 
be used for any of the following purposes? Please select just one answer for each 
potential use. If your state both permits and requires that evaluation results be used for a 
particular purpose, please report that purpose as required, a. To guide decisions about 
professional development and support for principals 
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Table C.4 (continued) 


Interview Questions Addressing the Teacher and Principal Certification 
Policy or Practice and Evaluation Policy or Practice 


Requiring principal TL89. Currently, does the state prohibit, permit, or require principal evaluation results to 
evaluations to inform be used for any of the following purposes? Please select just one answer for each 

decisions about potential use. If your state both permits and requires that evaluation results be used for a 

compensation particular purpose, please report that purpose as required, c. To guide decisions about 

principal salary increases; d. To guide decisions about principal bonuses or performance- 

based compensation other than salary 

Requiring principal TL89. Currently, does the state prohibit, permit, or require principal evaluation results to 
evaluations to inform be used for any of the following purposes? Please select just one answer for each 

decisions about career potential use. If your state both permits and requires that evaluation results be used for a 

advancement particular purpose, please report that purpose as required, e. To guide decisions about 

career advancement opportunities 

TL89. Currently, does the state prohibit, permit, or require principal evaluation results to 
be used for any of the following purposes? Please select just one answer for each 
potential use. If your state both permits and requires that evaluation results be used fora 
particular purpose, please report that purpose as required, b. To guide decisions about 

dismissal of principals. 

RTT application criterion: 

Ensure the equitable distribution of teachers and principals by developing a plan to ensure that 
students in high-poverty or high-minority schools have equitable access to highly effective 
teachers and principals and are not served by ineffective teachers and principals at higher rates 
than other students 


Requiring principal 
evaluations to inform 
decisions about 
dismissal of principals 


Requiring districts or 
schools to use 
strategies to promote a 
more equitable 
distribution of effective 
teachers 


Conducting analyses 
of teacher 

effectiveness based on 
student achievement 
growth to determine 
whether there has 
been a shift in the 
distribution of effective 

teachers 

Requiring districts or 
schools to use 
strategies to promote a 
more equitable 
distribution of effective 
principals 


TL68. Currently, does your state require that districts and/or schools use any of the 
following strategies aimed at promoting a more equitable distribution of effective teachers 
between high-poverty or high-minority schools and low-poverty or low-minority schools? a. 
Offer more compensation for staff who teach at high-poverty or high-minority schools; b. 
Offer performance-based compensation for effective teachers who teach at high-poverty 
or high-minority schools; c. Hire staff in high-poverty or high-minority schools and districts 
earlier; d. Increase external recruitment activities such as open houses, job fairs, 
presentations, or advertisements; e. Improve teaching and learning environments at high- 
poverty or high-minority schools such as improved opportunities for collaboration, 
improved school leadership, or improved facility quality; f. Offer more professional 
development, mentoring, or induction for teachers working at high-poverty or high-minority 

schools 

TL69. Within the past year, has your state conducted analyses to determine whether there 
has been a shift in the distribution of effective teachers between high-poverty or high- 
minority schools and low-poverty or low-minority schools? 

TL70. Which of the following factors were examined in these analyses? a. Teacher 
effectiveness based on student growth such as average teacher effectiveness ratings in 
schools, proportion of teachers rated as highly effective in schools, or the distribution of 
teachers by rating categories in schools 


TL94. Currently, does your state require that districts and/or schools use any of the 
following strategies aimed at promoting a more equitable distribution of effective principals 
between high-poverty or high-minority schools and low-poverty or low-minority schools? a. 
Offer more compensation for principals who work at high-poverty or high-minority schools; 
b. Offer performance-based compensation for effective principals of high-poverty or high- 
minority schools; c. Hire staff in high-poverty or high-minority schools and districts earlier; 
d. Increase external recruitment activities such as open houses, job fairs, presentations, 
or advertisements; e. Improve teaching and learning environments at high-poverty or high- 
minority schools such as improved opportunities for collaboration, improved school 
leadership, or improved facility quality; f. Offer more professional development for 
principals working at high-poverty or high-minority schools, such as training, coaching, or 
principal academies 
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Table C.4 (continued) 


Policy or Practice 

Interview Questions Addressing the Teacher and Principal Certification 
and Evaluation Policy or Practice 

Conducting analyses 
of principal 

effectiveness based on 
student achievement 
growth to determine 
whether there has 
been a shift in the 
distribution of effective 
principals 

TL95. Within the past year, has your state conducted analyses to determine whether there 
has been a shift in distribution of effective principals between high-poverty or high-minority 
schools and low-poverty or low-minority schools? 

TL96. Which of the following factors were examined in these analyses? b. Principal 
effectiveness ratings based on student growth 

RTT application criteria: 

Link student achievement and student growth data to the students’ teachers and principals and in- 
state teacher and principal preparation programs and publicly report the data for each credentialing 
program in the state 

Expand preparation and credentialing options and programs that are successful at producing 
effective teachers and principals 

Using teacher 
effectiveness ratings 
(based in part on 
student achievement 
growth) to assess the 
effectiveness of 
teacher certification 
programs 

TL1 1 . Currently, which of the following types of information does the state use to assess 
the effectiveness of teacher certification programs? Please indicate if each type of 
information is used for assessing effectiveness of traditional certification programs only, 
alternative certification programs only, both traditional and alternative programs, or 
neither, d. The effectiveness ratings (based in part on student growth) of credentialed 
teachers from each program 

Publicly reporting 
results from its 
evaluations of teacher 
certification program 
effectiveness 

TL12. Within the past year, did the state use the results from its evaluations of teacher 
certification programs in any of the following ways? For each potential use, please 
indicate if it was used for traditional certification programs only, alternative certification 
programs only, both traditional and alternative programs, or neither, f. Publicly reported 
results for each program 

Using results from its 
evaluations of teacher 
certification programs 
to provide additional 
funds for, expand, or 
promote certification 
programs that were 
shown to be effective 

TL12. Within the past year, did the state use the results from its evaluations of teacher 
certification programs in any of the following ways? For each potential use, please 
indicate if it was used for traditional certification programs only, alternative certification 
programs only, both traditional and alternative programs, or neither, b. Provided additional 
state funding to certification programs that were shown to be effective; c. Expanded 
and/or promoted certification programs that were shown to be effective 

Using principal 
effectiveness ratings 
(based in part on 
student achievement 
growth) to assess the 
effectiveness of 
principal certification 
programs 

TL28. Currently, which of the following types of information does the state use to assess 
the effectiveness of principal certification programs? Please indicate if each type of 
information is used for assessing effectiveness of traditional certification programs only, 
alternative certification programs only, both traditional and alternative programs, or 
neither, d. The effectiveness ratings (based in part on student growth) of credentialed 
principals from each program 

Publicly reporting 
results from its 
evaluations of principal 
certification program 
effectiveness 

TL29. Within the past year, did the state use the results from its evaluations of principal 
certification programs in any of the following ways? For each potential use, please 
indicate if it was used for traditional certification programs only, alternative certification 
programs only, both traditional and alternative programs, or neither, f. Publicly reported 
results for each program 

Using results from its 
evaluations of principal 
certification programs 
to provide additional 
funds, expand, or 
promote certification 
programs that were 
shown to be effective 

TL29. Within the past year, did the state use the results from its evaluations of principal 
certification programs in any of the following ways? For each potential use, please 
indicate if it was used for traditional certification programs only, alternative certification 
programs only, both traditional and alternative programs, or neither, b. Provided additional 
state funding to certification programs that were shown to be effective c. Expanded and/or 
promoted certification programs that were shown to be effective 


Source: Interviews with state administrators in spring 2012. 
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Table C.5. Interview Questions Addressing the School Turnaround (TA) Topic Area Policies and Practices 


Policy or Practice 

Interview Questions Addressing the School Turnaround Policy or Practice 

RTT application criterion: 

Intervene directly in the lowest-achieving schools and LEAs that are in improvement or corrective 
action status 

Having the authority 
to take over failing 
schools 

TA43. Does your state currently have the authority to take over failing schools? 

RTT application criterion: 

Support LEAs whose lowest-achieving schools are implementing one of the four school 
intervention models: turnaround model, restart model, school closure, or transformation model 

Providing training to 
the lowest-achieving 
schools or LEAs on 
analyzing student 
assessment data to 
improve instruction 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or the districts in which these schools are 
located? Please report technical assistance provided directly by state Department of 
Education staff as well as technical assistance funded by the state but provided by 
someone other than state Department of Education staff, for example, an external 
consultant or staff from a regional office, a. Training on analyzing student assessment data 
to improve instruction 

Helping the lowest- 
achieving schools or 
LEAs align curricula 
to state standards 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or the districts in which these schools are 
located? e. Help aligning school curricula to state standards 

Providing training to 
the lowest-achieving 
schools or LEAs on 
identifying and 
implementing 
effective curricula, 
instructional 
strategies, or school 
intervention models, 
or developing and 
implementing a 
school improvement 
plan 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or the districts in which these schools are 
located? c. Training on developing and implementing a school improvement plan; d. 
Training on identifying and implementing effective curricula, instructional strategies, or 
school intervention models 

Providing training to 
the lowest-achieving 
schools or LEAs on 
identifying and 
implementing 
strategies to address 
the needs of ELLs 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or the districts in which these schools are 
located? f. Training on identifying and implementing strategies to address the needs of 
English language learners 

Providing technical 
assistance to the 
lowest-achieving 
schools or LEAs on 
improving the quality 
of professional 
development 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or the districts in which these schools are 
located? g. Assistance on improving the quality of professional development 
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Table C.5 (continued) 


Interview Questions Addressing the School Turnaround Policy or Practice 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or the districts in which these schools are 
located? h. Assistance on analyzing and revising budgets to use resources more effectively 

TA22. Do the state’s current teacher assignment laws or policies for persistently lowest- 
achieving schools include any of the following features? e. School discretion or authority to 
decide which staff to hire for the persistently lowest-achieving schools 

TA17. Does the state currently provide persistently lowest-achieving schools any flexibility 
with, or exemptions from, collective bargaining agreements or policies that guide staffing in 
your state’s schools? 

TA25. Thinking specifically about your state’s persistently lowest-achieving schools, does 
the state, the districts, or individual schools currently have primary responsibility for the 
following? a. Setting student discipline policies; b. Developing school budgets; c. 
Establishing the curriculum including core texts; d. Setting student assessment policies 
except state mandated tests; e. Staff hiring, discipline, and dismissal; f. Determining the 
length of the school day; g. Determining the length of the school year; h. Setting 

requirements for professional development 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or the districts in which these schools are 
located? i. Assistance on developing strategies to recruit and retain more effective teachers 

TA22. Do the state’s current teacher assignment laws or policies for persistently lowest- 
achieving schools include any of the following features? a. Financial incentives for teachers 
to begin or continue to work in the state’s persistently lowest-achieving schools 


Implementing or 
providing technical 
assistance on 
strategies to recruit 
and retain effective 
teachers (such as 
financial incentives) 


Policy or Practice 

Providing operational 
flexibility and support 
to lowest-achieving 
schools or LEAs with 
regard to staffing and 
budgeting 


Having teacher 
tenure rules that 
affect placement in or 
removal from the 
lowest-achieving 
schools 


TA22. Do the state’s current teacher assignment laws or policies for persistently lowest- 
achieving schools include any of the following features? b. Other retention or recruitment 
efforts specifically targeted at the state’s persistently lowest-achieving schools (SPECIFY) 
TA22. Do the state’s current teacher assignment laws or policies for persistently lowest- 
achieving schools include any of the following features? f. Teacher tenure rules that affect 
placement in or removal from the persistently lowest-achieving schools 


Having state-level TA13. Does the state currently have any organizational or administrative structures 
administrative specifically intended to improve state capacity to support school turnaround efforts? 

structures intended to 

support school TA32. Which of the following groups are responsible for monitoring the state’s persistently 

turnaround efforts lowest-achieving schools? a. State department of education 

TA37. Does the state provide additional reporting or support for persistently lowest- 
achieving schools that miss improvement targets? 

Source: Interviews with state administrators in spring 2012. 
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Table C.6. Interview Questions Addressing the Charter Schools (CH) Topic Area Policies and Practices 


Policy or Practice 

Interview Questions Addressing the Charter School Policy or Practice 

RTT application criterion: 

Have a charter school law that does not prohibit or effectively inhibit increasing the number of 
high-performing charter schools in the state 

Having no restrictions on 
the creation of new 
charter schools or 
charter enrollment 

CH3. Currently, does the state have any restrictions on the creation of new charter 
schools and/or charter enrollment? 

RTT application criterion: 

Have laws, statutes, regulations, or guidelines regarding how charter school authorizers approve, 
monitor, hold accountable, reauthorize, and close charter schools 

In considering 
applications for new 
charter schools, giving 
priority to schools that 
propose to address 
needs of or target ELLs 

CH9. In considering applications for new charter schools, does your state give priority to 
schools that specifically propose to address the needs of and/or explicitly target English 
language learners? 

Monitoring the academic 
performance of charter 
schools 

CHI 3. On which of the following dimensions is charter school performance currently 
monitored by the state or its agent(s)? a. Academic performance 

Monitoring the non- 
academic performance 
of charter schools 

CHI 3. On which of the following dimensions is charter school performance currently 
monitored by the state or its agent(s)? b. Enrollment; c. Proportion of different student 
populations served such as English language learners; d. Finances and budget; e. 
Facilities 


Source: Interviews with state administrators in spring 2012. 
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ADDITIONAL INFORMATION ABOUT ENGLISH LANGUAGE LEARNER-FOCUSED 
ANALYSES FOR THE RTT COMPONENT OF THE EVALUATION 



Appendix D. Additional Information About ELL-Focused Analyses for RTT Usage of Policies and Practices Promoted by RTT 


This appendix contains additional information that is directly related to the English language 
learner (ELL)-focused analyses presented in Chapter V. In particular, this appendix lists the 
interview questions that address the ELL-focused policies and practices aligned with the Race to 
the Top (RTT) application criteria (Table D.l). 


Table D.l. Interview Questions Addressing the ELL-Focused Policies and Practices 


ELL-Focused 
Policy or Practice 
Prioritizing the 
adoption and 
implementation of 
supports to ELLs 


Providing targeted 
support or working 
with intermediaries 
to provide support 
to ELLs 


Implementing 
organizational or 
administrative 
changes to improve 
capacity to support 
ELLs 


Supporting districts 
or schools in 
implementing the 
Common Core 
State Standards 
with ELLs 
State longitudinal 
data system has 
program 
participation 
information about 
ELLs 


State longitudinal 
data system is 
used to inform and 
engage 

stakeholders and 
support decision 
makers in 
continuous 
improvement 
efforts for ELLs 


Interview Questions 

SC3. To what extent (great extent, moderate extent, little extent, or not at all) is each of the 
following educational reform initiatives a priority in your state for the current school year? I. 
Providing supports to ELLs 

SC21 . Which of the following statements best characterizes how ELLs fit into your state’s 
current education reform priorities? 1- ELLs are an explicit, central priority of statewide 
reform efforts, 2- English language learners are an emerging priority of statewide reform 
efforts, 3- Statewide reform efforts are designed to address the needs of all students, 
including ELLs 

SC8.T0 which of the following types of districts and/or schools has the state provided 
targeted support for implementation of statewide education reforms? By targeted supports, 
we mean supports that are provided to a specific group so schools and/or districts for a 
particular purpose, c. Districts and/or schools with high proportions of ELLs 

SCI 3. Does your state education agency currently work with any intermediaries to support 
the implementation of statewide education reform priorities in any of the following areas? 
For each reform area selected, please briefly describe the role assigned to intermediaries in 
that area. e. Providing supports for ELLs 

SC22. Since July 2008, has the state education agency implemented any of the following 
organizational or administrative changes to improve its capacity to address the needs of 
ELLs? a. Increased number of state Department of Education staff with ELL expertise b. 
Reorganized the structure of existing offices or department with ELL responsibility or 
increased the collaboration of staff across these offices or departments c. Increased use of 
external consultants with ELL expertise d. Increased ELL expertise within regional offices e. 
Redefined specific policies to better meet the needs of ELLs 
SA25. Since the adoption of [the Common Core State Standards/your current state 
standards], has the state provided any supports to districts and/or schools specifically 
designed to aid in the implementation of the state’s standards with ELL? This might include, 
but is not limited to, funds, professional development, technical assistance, or materials to 
support use of standards with ELLs 


DAI 6. Which of the following types of data specifically related to ELL students are currently 
included in your state’s longitudinal data system(s)? a. Students currently identified as ELLs 
b. Students formerly identified as ELLs c. Students’ English language proficiency test scores 
d. ELLs’ proficiency in their native (home) language e. Student participation in bilingual 
education programs, defined as programs that focus on developing literacy in two 
languages, which include non-English speakers and may include English speakers, and 
which involve instruction in English and another language f. Student participation in other 
educational programs specifically designed for ELLs g. Student’s native language h. 

Number of years the student has lived in U.S. k. The number of years that students have 
received English language instructional services I. The length of time it took for an ELL to 
exit ELL status 

DAI 7. For which of the following purposes are data in the state longitudinal data system 
specifically related to ELLs currently used by state-level staff? a. To track the progress of 
current ELLs b. To track the progress of former ELLs c. To track the ELL status of students 
d. To identify the needs of specific ELL populations such as refugees, migrant education 
students, or students who speak low-incidence languages e. To identify the professional 
development needs of teachers of ELLs f. To assess teacher effectiveness with ELLs g. To 
provide information to the parents of ELLs about their children’s progress h. To provide 
information to ELLs about their own progress i. To make decisions about exiting students 
from ELL status 
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Table D.1 (continued) 


ELL-Focused 
Policy or Practice 

Interview Questions 

Providing funding, 
materials, training, 
technical 

assistance, or other 
supports to districts 
to aid in the use of 
ELL-related data 

DAI 8. Within the past year, has the state provided funding, materials, training, technical 
assistance, or other supports to districts specifically designed to aid in the use of data 
related to ELLs? 

Teacher 

assignment laws or 
policies include 
financial incentives 
to recruit and retain 
teachers with ELL- 
expertise 

TA22. Do the state’s current teacher assignment laws or policies for persistently lowest- 
achieving schools include any of the following features? c. Financial incentives for staff with 
ELL expertise to begin or continue to work in persistently lowest-achieving schools d. Other 
retention or recruitment efforts targeted toward increasing the number of staff with ELL 
expertise in the persistently lowest-achieving schools (SPECIFY) 

Providing training 
to the lowest- 
achieving schools 
or local education 
agencies on 
identifying and 
implementing 
strategies to 
address the need 
of ELLs 

TA29. Which of the following types of training and/or technical assistance has the state 
provided in the current school year to support the improvement efforts of the persistently 
lowest-achieving schools in your state and/or district in which these schools are located? f. 
Training on identifying and implementing strategies to address the needs of ELLs 

Having state-level 
staff or consultants 
to support 
turnaround schools 
and districts 
working with ELLs 

TAM. Which of the following organizational or administrative structures are in place in your 
stat for this purpose? f. State-level staff or consultants to provide support to turnaround 
schools and districts in working with ELLs 

In considering 
applications for 
new charter 
schools, giving 
priority to schools 
that propose to 
address needs of 
or target ELLs 

CH9. In considering applications for new charter schools, does your state give priority to 
schools that specifically propose to address the needs of and/or explicitly target ELLs? 

Monitoring charter 
school performance 
based on the 
student populations 
(such as ELLs) 
served 

CHI 3. On which of the following dimensions is charter school performance currently 
monitored by the state or its agent(s)? c. Proportion of different student populations served 
such as ELLs 


Source: Interviews with state administrators in spring 2012. 

ELL= English language learner. 
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